US 12,393,601 B2
Distributed data processing using embedded hermetic and deterministic language
Timothée Peignier, Vancouver (CA); and Edward Steel, Vancouver (CA)
Assigned to Treasure Data, Inc., Mountain View, CA (US)
Filed by Treasure Data, Inc., Mountain View, CA (US)
Filed on Apr. 17, 2024, as Appl. No. 18/638,146.
Application 18/638,146 is a continuation of application No. 17/815,234, filed on Jul. 27, 2022, granted, now 12,013,867.
Prior Publication US 2024/0265026 A1, Aug. 8, 2024
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 16/25 (2019.01); G06F 8/30 (2018.01); G06F 8/70 (2018.01)
CPC G06F 16/252 (2019.01) [G06F 8/31 (2013.01); G06F 8/70 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A computer system comprising:
one or more central processing units (CPUs) that are communicatively coupled to a system clock, one or more network interfaces, and one or more database interfaces;
digital electronic main memory that is communicatively coupled to the one or more CPUs and storing one or more sequences of stored program instructions which, when executed using the one or more CPUs, cause the one or more CPUs to execute a plurality of different consumer services of a SaaS-based data analytics platform, each of the consumer services hosting an instance of a sandboxed runtime for a hermetic and deterministic programming language;
user function storage that is communicatively coupled to one of the database interfaces and storing a plurality of different user functions, each of the user functions having been programmed using the hermetic and deterministic programming language, the hermetic and deterministic programming language preventing each of the user functions from accessing a file system, the one or more network interfaces, and the system clock, each of the user functions being stored in association with a reference to a destination table of a destination database;
each of the consumer services being programmed to initiate a data ingestion process, load a copy of a user function from the user function storage to the sandboxed runtime that is associated with a particular consumer service from the plurality of different consumer services, using the sandboxed runtime local to the particular consumer service, execute the user function over records directed to the destination table identified in the reference, and filter the records or write new records resulting from the function to the destination table;
the consumer services being further programmed to receive an identifier of a source dataset for ingestion; read successive records of the source dataset until all records have been read; and for each record of the successive records that is read, determine whether a user function is present in the user function storage and has a reference to the destination table, and only when the user function is present and has the reference to the destination table, execute the user function over the each record.