US 12,314,254 B2
Batch data ingestion in database systems
Jiansheng Huang, San Mateo, CA (US); Jiaxing Liang, Bellevue, WA (US); Scott Ziegler, San Mateo, CA (US); Haowei Yu, Newark, CA (US); Benoit Dageville, San Mateo, CA (US); and Varun Ganesh, San Bruno, CA (US)
Assigned to Snowflake Inc., Bozeman, MT (US)
Filed by Snowflake Inc., Bozeman, MT (US)
Filed on Feb. 25, 2022, as Appl. No. 17/652,504.
Application 17/652,504 is a continuation of application No. 16/365,219, filed on Mar. 26, 2019, granted, now 11,294,890.
Application 16/365,219 is a continuation of application No. 16/201,854, filed on Nov. 27, 2018, granted, now 11,055,280.
Claims priority of provisional application 62/591,118, filed on Nov. 27, 2017.
Prior Publication US 2022/0179846 A1, Jun. 9, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 16/23 (2019.01); G06F 16/22 (2019.01); G06F 16/25 (2019.01)
CPC G06F 16/2386 (2019.01) [G06F 16/2255 (2019.01); G06F 16/2282 (2019.01); G06F 16/254 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
polling, by at least one hardware processor, a first file queue to determine whether any new files have been committed to the first file queue since a last time the first file queue was polled;
accessing a data storage device to detect whether a new file has been added to the data storage device;
determining that the new file comprising new file data has been added to a database source comprising the data storage device;
processing information associated with the new file to retrieve a table identifier for an individual target table corresponding to the new file automatically in response to determining that the new file has been added to the database source comprising the data storage device;
in response to determining that the individual target table corresponding to the table identifier represents a first target table, assigning a pipe to the new file for the pipe to retrieve and direct the new file data to the first target table;
without requiring a specific command from a user of a first account and in response to determining that the new file has been added:
dividing the new file into a plurality of partitions; and
incrementally inserting each of the plurality of partitions into a respective micro-partition of a plurality of micro-partitions of the first target table partition-by-partition; and
providing a second file queue that corresponds to a second target table, the second target table different than the first target table associated with the first file queue, the first file queue corresponding to the first account and the second file queue corresponding to a second account, the first target table corresponding to the first account being isolated from the second target table corresponding to the second account.