US 11,055,262 B1
Extensible streams on data sources
Istvan Cseri, Seattle, WA (US); Tyler Jones, San Francisco, CA (US); Dinesh Chandrakant Kulkarni, Sammamish, WA (US); and Igor Zinkovsky, San Mateo, CA (US)
Assigned to Snowflake Inc., San Mateo, CA (US)
Filed by Snowflake Inc., San Mateo, CA (US)
Filed on Mar. 9, 2020, as Appl. No. 16/812,940.
Int. Cl. G06F 16/18 (2019.01); G06F 16/182 (2019.01); G06F 16/2453 (2019.01); G06F 16/2455 (2019.01); G06F 16/23 (2019.01)
CPC G06F 16/1865 (2019.01) [G06F 16/1824 (2019.01); G06F 16/2322 (2019.01); G06F 16/24532 (2019.01); G06F 16/24568 (2019.01)] 31 Claims
OG exemplary drawing
1. A network-based database system comprising:
at least one hardware processor; and
a memory storing instructions that cause the at least one hardware processor to perform operations comprising:
determining a set of shards corresponding to an external data source accessible via a network, the external data source being hosted by an external system separate from the network-based database system;
determining, using a stream object provided by the network-based database system separate from the external system, a set of offsets of each shard of the set of shards, the set of offsets being determined by executing a user defined function (UDF) included in the stream object, the stream object and the UDF being implemented in a particular programming language different than a database query language;
identifying an operation to perform on the set of shards, the operation comprising a write operation;
based on the set of shards and the set of offsets, performing, using the UDF included in the stream object the write operation on the external data source hosted by the external system; and
incrementing a property indicating a different version of the stream object after performing the write operation on the external data source, the property being included in the stream object.