US 11,841,871 B2
Managing extract, transform and load systems
Chengxuan Xing, Romsey (GB); Doina Liliana Klinger, Winchester (GB); Alexander Robert Wood, Romsey (GB); and Tom Soal, Whitehill (GB)
Assigned to International Business Machines Corporation, Armonk, NY (US)
Filed by International Business Machines Corporation, Armonk, NY (US)
Filed on Jun. 29, 2021, as Appl. No. 17/362,157.
Prior Publication US 2022/0414113 A1, Dec. 29, 2022
Int. Cl. G06F 17/00 (2019.01); G06F 16/25 (2019.01); G06F 11/34 (2006.01); G06F 16/215 (2019.01); G06F 11/14 (2006.01)
CPC G06F 16/254 (2019.01) [G06F 11/1402 (2013.01); G06F 11/3414 (2013.01); G06F 16/215 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method for extracting, transforming and loading data between clients, servers and storage locations, the computer-implemented method comprising:
extracting, by one or more processors, units of data from a source system;
queueing, by the one or more processors, the units of data to a queue for processing;
processing, by the one or more processors, the units of data from the queue, based on transforming the units of data and loading the units of data to a target system, wherein the units of data are subject to a rate limit error from the loading and are requeued in the queue for reprocessing;
monitoring the queue and detecting if there are more than a threshold number of requeued units of data in the queue; and
responsive to more than a threshold number of requeued units of data in the queue, imposing, by the one or more processors, at least one restriction on the extracting and applying a retry schedule for reprocessing the requeued units of data.