US 12,271,376 B1
Generating metadata from a scan of a data object in an object store for performing subsequent queries to the data object
Ning Liao, Sammamish, WA (US); Fusheng Yuan, Kenmore, WA (US); and Kaiwen Qu, Seattle, WA (US)
Assigned to Amazon Technologies, Inc., Seattle, WA (US)
Filed by Amazon Technologies, Inc., Seattle, WA (US)
Filed on May 18, 2018, as Appl. No. 15/983,942.
Int. Cl. G06F 16/00 (2019.01); G06F 9/54 (2006.01); G06F 16/2453 (2019.01); G06F 16/9535 (2019.01)
CPC G06F 16/24539 (2019.01) [G06F 9/547 (2013.01); G06F 16/24537 (2019.01); G06F 16/9535 (2019.01)] 19 Claims
OG exemplary drawing
 
1. A system, comprising:
an object data store, comprising:
one or more computing devices, respectively comprising a processor and a memory; and
one or more storage devices that store a data object as unstructured data, wherein the unstructured data comprises a byte sequence without storing information indicating a structure used to access the data object and the data object includes a range of records;
wherein the object data store is configured to:
receive a request to scan the unstructured data of the data object in the one or more storage devices;
generate metadata for the data object that provides structural information for the unstructured data of the data object, wherein the metadata comprises respective offsets for a plurality of records of the byte sequence, the respective offsets determined according to respective numbers of bytes determined from parsing from one record delimiter character to another in the byte sequence based on the scanned the unstructured data of the data object;
store the generated metadata of structural information for the unstructured data of the data object in a metadata store;
receive a request to perform a query on a range of records of the data object, and responsive to receiving the request:
access the stored metadata at the metadata store to generate a plan to perform the requested query, the plan generated in accordance with the respective offsets for the plurality of records in the stored metadata;
perform the query on the range of records of the data object in the one or more storage devices according to the generated plan; and
return a result of the query.