US 11,841,845 B2
Data consistency mechanism for hybrid data processing
Jianjun Chen, Los Angeles, CA (US); Yonghua Ding, Los Angeles, CA (US); Ye Liu, Los Angeles, CA (US); Fangshi Li, Los Angeles, CA (US); Lixun Cao, Beijing (CN); Yang Liu, Beijing (CN); Li Zhang, Los Angeles, CA (US); Mingyi Zhang, Los Angeles, CA (US); Xiangrui Meng, Beijing (CN); Junda Zhao, Beijing (CN); Lei Zhang, Beijing (CN); and Rui Shi, Beijing (CN)
Assigned to LEMON INC., Grand Cayman (KY)
Filed by LEMON INC., Grand Cayman (KY)
Filed on Aug. 31, 2021, as Appl. No. 17/462,938.
Prior Publication US 2023/0062198 A1, Mar. 2, 2023
Int. Cl. G06F 16/23 (2019.01); G06F 16/27 (2019.01); G06F 16/28 (2019.01)
CPC G06F 16/2365 (2019.01) [G06F 16/2358 (2019.01); G06F 16/278 (2019.01); G06F 16/283 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A system, comprising:
at least one processor; and
at least one memory communicatively coupled to the at least one processor and comprising instructions that upon execution by the at least one processor cause the system to perform operations comprising:
generating logical logs and log sequence numbers (LSNs) associated with the logical logs based on data captured by a first processing engine, the first processing engine configured to perform online transactional processing, wherein the LSNs comprises data definition language (DDL) LSNs associated with DDL operations;
propagating the logical logs and the LSNs to a storage, the storage configured to be in communication with the first processing engine and a second processing engine, the second processing engine configured to perform online analytical processing, wherein the storage is divided into a first part and a second part, and wherein the second part of the storage comprises partitions each of which comprises a delta store and a base store;
storing and distributing the LSNs by a metadata service, wherein the metadata service is configured to periodically pull DDL information from the first processing engine and identify a newest DDL LSN as a current metadata version; and
wherein the first processing engine, the second processing engine, the storage, and the metadata service are modularized, are configured to be decoupled from each other, and support the LSNs for maintaining data consistency across the system.