US 11,940,881 B1
Efficient identification of snapshot objects using full and differential snapshot manifests
Swapnil Srivastava, Seattle, WA (US); and Ravi Sekhar Cherukuri, Kirkland, WA (US)
Assigned to Amazon Technologies, Inc., Seattle, WA (US)
Filed by Amazon Technologies, Inc., Seattle, WA (US)
Filed on Sep. 21, 2020, as Appl. No. 17/027,374.
Int. Cl. G06F 11/14 (2006.01); G06F 16/11 (2019.01)
CPC G06F 11/1453 (2013.01) [G06F 11/1451 (2013.01); G06F 16/128 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A system for conducting post-processing of a plurality of object-based snapshots of at least two distinct block-storage volumes, each object-based snapshot representing data of a corresponding block-storage volume at a point-in-time, the system comprising:
one or more hardware storage devices comprising:
a metadata store storing metadata identifying individual object-based snapshots of the plurality of object-based snapshots as incremental or non-incremental, wherein an incremental object-based snapshot is associated with a parent object-based snapshot that represents an earlier state of the block-storage volume corresponding to the incremental object-based snapshot, and wherein a non-incremental object-based snapshot is not associated with a parent object-based snapshot; and
an object data store including:
snapshot objects, each snapshot object representing one or more blocks of data;
full manifests, each full manifest corresponding to an object-based snapshot of the plurality of object-based snapshots and including a full listing of snapshot objects within the data store that collectively represent data of the corresponding block-storage volume at the point-in-time of the object-based snapshot; and
one or more differential manifests, each differential manifest corresponding to an incremental object-based snapshot of the plurality of object-based snapshots that depends on a prior object-based snapshot, and including a listing of snapshot objects unique to the incremental object-based snapshot relative to the prior object-based snapshot; and
one or more processors configured to:
generate a listing identifying those snapshot objects, within the data store including snapshot objects, that are referenced by any one or more of the plurality of object-based snapshots by:
identifying from the metadata a first set of object-based snapshots corresponding to any object-based snapshots, within the plurality of object-based snapshots, that are non-incremental;
retrieving from the object data store a set of full manifests corresponding to the first set of object-based snapshots, corresponding to any object-based snapshots, within the plurality of object-based snapshots, that are non-incremental;
identifying from the metadata a second set of object-based snapshots corresponding to any object-based snapshots, within the plurality of object-based snapshots, that are incremental;
retrieving from the object data store a set of differential manifests corresponding to the second set of object-based snapshots, corresponding to any object-based snapshots, within the plurality of object-based snapshots, that are incremental; and
aggregating snapshot objects listed within the set of full manifests, corresponding to any object-based snapshots, within the plurality of object-based snapshots, that are non-incremental, with snapshot objects listed within the set of differential manifests, corresponding to any object-based snapshots, within the plurality of object-based snapshots, that are incremental, to result in the listing identifying those snapshot objects, within the data store including snapshot objects, that are referenced by any one or more of the plurality of object-based snapshots; and
conduct post-processing of the object-based snapshots by use of the listing identifying those snapshot objects within the object data store that are referenced by any one or more of the object-based snapshots.