US 12,468,707 B1
Down-sized cluster performance modeling for a tiered data processing service
Induja Sreekanthan, Mountain View, CA (US); Sriram Subramanian, Mountain View, CA (US); Venu Gopal Nayar, San Jose, CA (US); Athanasios Papathanasiou, Los Gatos, CA (US); and Vijayan Prabhakaran, San Francisco, CA (US)
Assigned to Amazon Technologies, Inc., Seattle, WA (US)
Filed by Amazon Technologies, Inc., Seattle, WA (US)
Filed on Mar. 31, 2022, as Appl. No. 17/710,459.
Int. Cl. G06F 16/2455 (2019.01); G06F 16/21 (2019.01); G06F 16/2457 (2019.01); G06F 16/28 (2019.01)
CPC G06F 16/24554 (2019.01) [G06F 16/219 (2019.01); G06F 16/2457 (2019.01); G06F 16/285 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A system, comprising:
one or more computing devices configured to implement a data processing service, wherein the data processing service implements a plurality of storage tiers to store data blocks, the plurality of storage tiers comprising at least:
a first storage tier implemented on storage included in nodes of a first node cluster of the data processing service, wherein the first node cluster has a first storage capacity to store a first quantity of data blocks, wherein more recently accessed data blocks are stored in the first storage tier; and
a second storage tier implemented on storage of a data storage service, wherein less recently accessed data blocks are stored in the second storage tier; and
wherein the data processing service is further configured to:
maintain metadata comprising block identifiers for the first quantity of data blocks that fit within the first storage capacity of the first storage tier, wherein the block identifiers for the first quantity of data blocks that fit within the first storage capacity of the first storage tier comprise a sub-set of the block identifiers that would fit within a second storage capacity of a down-sized version of the first storage tier, and wherein a particular block identifier included in the sub-set of the block identifiers is evicted from the sub-set of the block identifiers in response to an eviction of a respective data block from the first storage tier; and
model based, at least in part, on a hit-count counter for the sub-set of the block identifiers, hypothetical performance of the down-sized version of the first storage tier having the second storage capacity, wherein modeling the hypothetical performance comprises:
maintaining the hit-count counter for the sub-set of the block identifiers; and
incrementing the hit-count counter, responsive to a query targeting a data block stored in the first storage tier and having a block identifier included in the sub-set of the block identifiers.