US 12,393,634 B1
Scalable graph-based vector storage and search in distributed storage systems
Hakan Ferhatosmanoglu, Stratford-upon-Avon (GB); Andrew Kutsy, London (GB); Jonathan S. Katz, New York, NY (US); Mrithyunjaya Kumar Annapragada, Concord, MA (US); Marc Brooker, Seattle, WA (US); Andrew Kent Warfield, Vancouver (CA); and Yu-Ju Huang, Cambridge (GB)
Assigned to Amazon Technologies, Inc., Seattle, WA (US)
Filed by Amazon Technologies, Inc., Seattle, WA (US)
Filed on Jun. 28, 2024, as Appl. No. 18/759,330.
Int. Cl. G06F 16/901 (2019.01); G06F 16/22 (2019.01)
CPC G06F 16/9024 (2019.01) [G06F 16/2237 (2019.01)] 16 Claims
OG exemplary drawing
 
1. A vector management system comprising:
a distributed object storage service comprising a plurality of object storage devices;
a processing service comprising one or more processors configured to execute specific instructions, the processing service configured to generate a set of vectors from a set of source data;
an index service comprising a set of one or more processors configured to execute specific instructions, the index service configured to:
generate an index of the set of vectors as a distributed hierarchical proximity-based graph data structure, wherein the distributed hierarchical proximity-based graph data structure comprises:
a different fine proximity-based graph data structures for each subset of a plurality of subsets of vectors of the set of vectors; and
a course proximity-based graph data structure comprising at least one representative vector for each subset of the plurality of subsets of vectors, wherein the course proximity-based graph data structure links the different fine proximity-based graph data structures to form the distributed hierarchical proximity-based graph data structure; and
store the set of vectors across the plurality of object storage devices of the distributed object storage service based on the distributed hierarchical proximity-based graph data structure, wherein a subset of the set of vectors is stored as a leaf object comprising a serialized version of a fine proximity-based graph data structure; and
a query service comprising a set of one or more processors configured to execute specific instructions, the query service configured to, in response to receiving a vector query:
load a plurality of subsets of the set of vectors from the distributed object storage service based on the distributed hierarchical proximity-based graph data structure;
determine, based on evaluation of similarities between a query vector and at least a portion of vectors in the distributed hierarchical proximity-based graph data structure, a plurality of objects to be obtained from the distributed object storage service; and
load a plurality of subsets of vectors from the distributed object storage service by loading the plurality of objects from the distributed object storage service at least partially in parallel, wherein a first object of the plurality of objects is obtained from a first subset of storage devices of the distributed object storage service and a second object of the plurality of objects is obtained from a second subset of storage devices of the distributed object storage service.