CPC H04L 67/1001 (2022.05) [G06F 9/5066 (2013.01); G06F 9/5083 (2013.01); G06F 9/5088 (2013.01); G06F 16/00 (2019.01); G06F 16/278 (2019.01); H04L 43/08 (2013.01); H04L 67/148 (2013.01); G06F 2209/5017 (2013.01)] | 20 Claims |
1. A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations comprising:
partitioning a data set for an application job into a plurality of partitions based on a key;
assigning, to each worker computer in a set of worker computers, one or more partitions of the plurality of partitions;
receiving, from each respective worker computer in the set of worker computers, a respective load measure indicating a computational load of the respective worker computer;
determining, using the respective load measure of each respective worker computer, a weight of a move operation for moving at least one partition from a first worker computer of the set of worker computers to a second worker computer of the set of worker computers;
determining that the weight of the move operation satisfies a threshold; and
in response to determining that the weight of the move operation satisfies the threshold, moving the at least one partition from the first worker computer of the set of worker computers to the second worker computer of the set of worker computers.
|