US 12,314,225 B2
Continuous ingestion of custom file formats
Tyler Arthur Akidau, Seattle, WA (US); Thierry Cruanes, San Mateo, CA (US); Benoit Dageville, San Mateo, CA (US); Ganeshan Ramachandran Iyer, Redmond, WA (US); and Subramanian Muralidhar, Mercer Island, WA (US)
Assigned to Snowflake Inc., Bozeman, MT (US)
Filed by Snowflake Inc., Bozeman, MT (US)
Filed on Oct. 27, 2022, as Appl. No. 18/050,122.
Prior Publication US 2024/0143548 A1, May 2, 2024
Int. Cl. G06F 15/16 (2006.01); G06F 9/50 (2006.01); G06F 16/11 (2019.01); G06F 16/14 (2019.01)
CPC G06F 16/148 (2019.01) [G06F 9/5022 (2013.01); G06F 16/116 (2019.01)] 12 Claims
OG exemplary drawing
 
1. A method comprising:
defining, by a multi-tenant data system comprising at least one hardware processor, a custom file format for use in auto-ingestion of files in the custom file format, the custom file format not being natively supported by the multi-tenant data system and includes an unstructured file type;
setting a user defined function (UDF) to parse files in the custom file format and return a table with one or more rows and a variant column;
managing a pool of execution nodes to be shared by a plurality of accounts in the multi-tenant data system;
receiving, from a first account, a first notification of a first file in a natively supported file format;
acquiring a first set of execution nodes from the pool of execution nodes;
assigning one or more ingests tasks related to ingestion of the first file to the first set of execution nodes, the first set of execution nodes being simultaneously assigned to at least one other account to perform at least one additional ingest task for the at least other account;
receiving, from a second account, a second notification of at a second file in the custom file format for auto-ingestion;
performing auto-ingestion of the second file using a pipe set for the custom file format executing the UDF, the UDF including a first argument including reference to a location of the second file and a second argument including scanner options defined for the custom file format, including:
acquiring a second set of execution nodes from the pool of execution nodes, the second set having different size execution nodes than the first set of execution nodes,
assigning, by the compute service manager, one or more ingest tasks related to second file,
pinning the second set of execution nodes such that the pinned execution nodes cannot simultaneously perform ingest tasks for other accounts in the multi-tenant data system except the second account, and
performing, by the pinned execution nodes, the one or more ingest tasks;
storing data from the at least one file in one or more rows in a source table in a format used by the data system;
registering the one or more rows in the source table;
committing the one or more rows in the source table; and
in response to committing the one or more rows, releasing the second set of execution nodes.