US 12,468,766 B2
Execution of a shuffle operator via a database system based on allocating memory units
Ellis Mihalko Saupe, University City, MO (US); and Andrew Park, St. Charles, IL (US)
Assigned to Ocient Holdings LLC, Chicago, IL (US)
Filed by Ocient Holdings LLC, Chicago, IL (US)
Filed on Nov. 13, 2024, as Appl. No. 18/945,889.
Application 18/945,889 is a continuation of application No. 18/226,525, filed on Jul. 26, 2023, granted, now 12,210,572.
Claims priority of provisional application 63/506,852, filed on Jun. 8, 2023.
Prior Publication US 2025/0068677 A1, Feb. 27, 2025
Int. Cl. G06F 16/903 (2019.01)
CPC G06F 16/90335 (2019.01) 20 Claims
OG exemplary drawing
 
18. A method for execution by at least one processor of a node of a database system, comprising:
determining a query for execution that includes a shuffle operator;
executing the shuffle operator in conjunction with executing the query collectively with a plurality of other nodes of the database system based on:
during a first temporal period:
reserving a first amount of memory for data to be received from the plurality of other nodes for processing in conjunction with executing the shuffle operator;
allocating, to each of the plurality of other nodes, a corresponding number of fixed-memory units based on the first amount of memory;
updating state data to indicate the corresponding number of fixed-memory units allocated to the each of the plurality of other nodes;
receiving first data from the plurality of other nodes in accordance with the corresponding number of fixed-memory units allocated to the each of the plurality of other nodes based on updating the state data;
processing the first data in accordance with execution of the shuffle operator;
during a second temporal period after the first temporal period:
updating the first amount of memory to a second amount of memory reserved for the data to be received from the plurality of other nodes for processing in conjunction with executing the shuffle operator based on comparing an available amount of memory with a configured memory threshold;
re-allocating, to the each of the plurality of other nodes, an updated corresponding number of fixed-memory units based on changing from the first amount of memory to the second amount of memory;
further updating the state data to indicate the updated corresponding number of fixed-memory units allocated to the each of the plurality of other nodes;
receiving second data from the plurality of other nodes in accordance with the updated corresponding number of fixed-memory units allocated to the each of the plurality of other nodes based on updating the state data; and
processing the second data in accordance with further execution of the shuffle operator.