US 12,143,502 B2
Data transfer acceleration via content-defined chunking
Stephen James Day, Kirkland, WA (US); Valient Gough, Kirkland, WA (US); Cheng Pan, Bellevue, WA (US); and Seth Alexander Bunce, Seattle, WA (US)
Assigned to GM Cruise Holdings LLC, San Francisco, CA (US)
Filed by GM Cruise Holdings LLC, San Francisco, CA (US)
Filed on Dec. 30, 2021, as Appl. No. 17/565,902.
Prior Publication US 2023/0216690 A1, Jul. 6, 2023
Int. Cl. H04L 9/32 (2006.01); H04L 9/06 (2006.01); H04L 9/14 (2006.01); H04L 9/34 (2006.01)
CPC H04L 9/3242 (2013.01) [H04L 9/0643 (2013.01); H04L 9/14 (2013.01); H04L 9/34 (2013.01)] 19 Claims
OG exemplary drawing
 
17. A computer-implemented method, comprising:
performing, at a worker node, a first simulation task using a first binary image, the first binary image having a data structure that includes autonomous vehicle sensor data for autonomous vehicle simulation;
dividing the first binary image into a plurality of variable-sized chunks, wherein the first binary image is an aggregate of a plurality of files, and wherein the dividing does not depend on file boundaries;
computing hashes of the variable-sized chunks, and storing the hashes in a content addressable storage (CAS) with the hashes as keys;
receiving a manifest comprising an ordered set of second hashes, wherein the second hashes are hashes of chunks of a second binary image to be downloaded and used in a second simulation task;
building a download plan, comprising searching the CAS for the second hashes, compiling a first group of hashes found in the CAS, and compiling a second group of hashes not found in the CAS, and ordering the first group of hashes and second set of hashes to comply with the manifest;
downloading chunks corresponding to the second group of hashes;
reordering the downloaded chunks and chunks that correspond to first group of hashes to build a copy of the second binary image; and
performing, at the worker node, the second simulation task using the copy of the second binary image.