US 12,393,562 B2
Data processing method and apparatus for columnar storage data warehouse
Zhaowei Huang, Beijing (CN); and Yuanpu Ding, Beijing (CN)
Assigned to Beijing Volcano Engine Technology Co., Ltd., Beijing (CN)
Filed by Beijing Volcano Engine Technology Co., Ltd., Beijing (CN)
Filed on Nov. 25, 2024, as Appl. No. 18/958,789.
Claims priority of application No. 202311667945.4 (CN), filed on Dec. 6, 2023.
Prior Publication US 2025/0190413 A1, Jun. 12, 2025
Int. Cl. G06F 16/22 (2019.01); G06F 16/23 (2019.01)
CPC G06F 16/221 (2019.01) [G06F 16/2343 (2019.01)] 16 Claims
OG exemplary drawing
 
1. A data processing method for a columnar storage data warehouse, wherein the columnar storage data warehouse is divided to comprise a key-value storage module and a columnar storage module, and the data processing method comprises:
receiving a data write request, wherein the data write request is used to request to write target data in a target write manner, and the target data comprises at least one row of data;
in response to the target write manner being a first write manner, writing, by using a key-value storage manner, the target data into the key-value storage module for storage, so that the target data is subsequently read from the key-value storage module, each row of data in the at least one row of data respectively corresponds to one key-value pair, and a data write amount corresponding to the first write manner is less than a first threshold;
or,
in response to the target write manner being a second write manner, writing, by using a columnar storage technology, the target data into the columnar storage module for storage, wherein a data write amount corresponding to the second write manner is greater than the data write amount corresponding to the first write manner;
wherein the writing, by using the key-value storage manner, the target data into the key-value storage module for storage comprises:
obtaining a first key-value object corresponding to a data lock in the key-value storage module, wherein a key of the first key-value object indicates that the first key-value object corresponds to the data lock, and a value of the first key-value object indicates a state of the data lock;
in response to the value of the first key-value object indicating that the data lock is in an unlocked state, modifying the value of the first key-value object to obtain a second key-value object, wherein a value of the second key-value object indicates that the data lock is in a locked state; and
writing, by using the key-value storage manner, the target data into the key-value storage module for storage;
wherein the key of the first key-value object indicates that the first key-value object corresponds to a data lock of a first data storage unit, and the value of the first key-value object further indicates a first row number of data that has been stored in the first data storage unit;
wherein the writing, by using the key-value storage manner, the target data into the key-value storage module for storage comprises writing the target data into the first data storage unit in the key-value storage module by using the key-value storage manner; and
wherein the value of the second key-value object further indicates a second row number of data stored in the first data storage unit after the target data is written into the first data storage unit.