US 12,229,137 B1
Checkpoint and restore based startup of executor nodes of a distributed computing engine for processing queries
Xinyang Ge, Kirkland, WA (US); Lixiang Ao, Sunnyvale, CA (US); Haonan Jing, San Jose, CA (US); and Aaron Daniel Davidson, Berkeley, CA (US)
Assigned to Databricks, Inc., San Francisco, CA (US)
Filed by Databricks, Inc., San Francisco, CA (US)
Filed on Jan. 12, 2024, as Appl. No. 18/412,438.
Int. Cl. G06F 16/2453 (2019.01)
CPC G06F 16/24549 (2019.01) [G06F 16/24542 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method, comprising:
starting execution of one or more processes for processing queries in a source executor node of a distributed computing engine configured for receiving and processing queries, the one or more processes including a runtime execution engine of a programming language used for implementing instructions for the processing queries;
sending the queries to the source executor node for execution of the queries on the source executor node;
performing checkpoint of the one or more processes executing on the source executor node to generate a checkpoint image, wherein the execution of the queries on the source executor node causes just-in-time compilation of at least a portion of code used for the processing queries, the just-in-time compilation causing generation of native instructions for the source executor node, wherein the native instructions are stored in the checkpoint image;
storing the checkpoint image in a checkpoint image repository in the source executor node;
providing the checkpoint image from the checkpoint image repository to a plurality of target executor nodes of the distributed computing engine;
for each target executor node of the plurality of target executor nodes of the distributed computing engine, performing restore of checkpoint image to restore a check point image on said each target executor node of the plurality of target executor nodes based on the native instructions of the provided checkpoint image; and
processing additional queries using the plurality of target executor nodes in the distributed computing engine to perform operations related to creation of data structures after the performing the restore of checkpoint image.