US 11,841,864 B2
Method, device, and program product for managing index of streaming data storage system
Pengfei Su, Shanghai (CN); and Lingling Yao, Shanghai (CN)
Assigned to EMC IP HOLDING COMPANY LLC, Hopkinton, MA (US)
Filed by EMC IP Holding Company LLC, Hopkinton, MA (US)
Filed on Oct. 10, 2022, as Appl. No. 18/045,300.
Application 18/045,300 is a continuation of application No. 17/121,653, filed on Dec. 14, 2020, granted, now 11,500,879.
Claims priority of application No. 202011149599.7 (CN), filed on Oct. 23, 2020.
Prior Publication US 2023/0061102 A1, Mar. 2, 2023
Int. Cl. G06F 17/00 (2019.01); G06F 7/00 (2006.01); G06F 16/2455 (2019.01); G06F 16/28 (2019.01); G06F 16/248 (2019.01); G06F 16/22 (2019.01)
CPC G06F 16/24568 (2019.01) [G06F 16/2255 (2019.01); G06F 16/248 (2019.01); G06F 16/283 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method, comprising:
partitioning, by a device comprising a processor, a first layer of a system index, associated with a streaming data storage system, into first shards of a first shard group, wherein the first shards store first object indexes of first data objects stored by the streaming data storage system according to a hash ring;
in response to determining that a state of the first shard group satisfies a defined expansion condition, creating, by the device, a second shard group in a second layer of the system index that is not the first layer, the second shard group comprising second shards that store second object indexes of second data objects that are designated to enter the streaming data storage system, wherein a first number of the second shards of the second shard group is an integer multiple of a second number of the first shards of the first shard group;
in response to the creating of the second shard group:
marking, by the device, the second layer of the system index as an active layer, and
marking, by the device, first layers of the system index, comprising the first layer, as frozen layers; and
managing, by the device, the streaming data storage system based on the second shards in the active layer and the first shards in the frozen layers, wherein the system index is associated with a layer group comprising the second layer and at least one of the first layers, and wherein the managing comprises, in response to a combined number of object indexes, comprising the first object indexes and the second object indexes and stored in the layer group, being determined to be lower than a threshold number, combining respective shards in the layer group.