US 12,292,853 B1
Object-based storage with garbage collection and data consolidation
Graham Edwin Ellis, Seattle, WA (US); Ying Fairweather, Sammamish, WA (US); Thorne Davis Garvin, Spokane, WA (US); Steven Henry Haber, Seattle, WA (US); Yuxi Bai, Seattle, WA (US); Michael Anthony Chmiel, Seattle, WA (US); Pathirat Kosakanchit, Seattle, WA (US); Jonathan Michael MacLaren, Seattle, WA (US); Matthew Christopher McMullan, Bellevue, WA (US); Tyler Morrison Moody, Seattle, WA (US); Aaron James Passey, San Rafael, CA (US); Rowan Arthur Phipps, Seattle, WA (US); and Thomas Gregory Rothschilds, Seattle, WA (US)
Assigned to Qumulo, Inc., Seattle, WA (US)
Filed by Qumulo, Inc., Seattle, WA (US)
Filed on Nov. 6, 2023, as Appl. No. 18/503,082.
Int. Cl. G06F 16/11 (2019.01); G06F 12/02 (2006.01)
CPC G06F 16/122 (2019.01) [G06F 12/0253 (2013.01); G06F 2212/702 (2013.01)] 28 Claims
OG exemplary drawing
 
1. A method for managing data in a file system over a network using one or more processors to execute instructions that are configured to cause performance of actions, comprising:
determining an object store for one or more write requests provided by a file system client based on the file system, wherein each write request includes one or more data blocks associated with a write transaction;
generating one or more write ahead log (WAL) entries that include the one or more data blocks, wherein one or more of a rule, instruction or a library is employed to determine one or more types of index data structures selected from a plurality of types of data structures for efficiently including the one or more data blocks with a WAL entry;
generating a WAL object based on the one or more WAL entries;
storing the WAL object in the object store;
updating an in-memory overlay with the one or more types of determined index data structures to associate the one or more data blocks with the WAL object;
deleting each WAL object in the object store that is unassociated with one or more data blocks and deleting each non-referenced data object in the object store; and
employing a threshold for a number of WAL objects stored in the object store to initiate a checkpoint operation that performs further actions, including:
generating an index object that includes one or more index entries that associate one or more other data blocks with one or more data objects stored in the object store;
updating the index object to include one or more index entries that associate the one or more data blocks with the WAL object;
storing the updated index object in the object store, wherein the WAL object is converted to a data object based on the WAL object being referenced in the updated index object;
updating the in-memory overlay to remove the association of the one or more data blocks and the WAL object; and
initiating garbage collection for one or more data objects stored in the object store based on one or more additional financial costs charged by a cloud computing provider of an application programming interface (API) for the file system due to one or more of an amount of stored data objects being above a predetermined amount for the object store or a total amount of data objects.