US 12,461,898 B2
Remediating a change in a system of representation of information in data used by a data pipeline
Ofir Ezrielev, Beer Sheva (IL); Hanna Yehuda, Acton, MA (US); and Inga Sogaard, Wichita, KS (US)
Assigned to Dell Products L.P., Round Rock, TX (US)
Filed by Dell Products L.P., Round Rock, TX (US)
Filed on Jun. 29, 2023, as Appl. No. 18/343,969.
Prior Publication US 2025/0004999 A1, Jan. 2, 2025
Int. Cl. G06F 16/21 (2019.01); G06F 16/2452 (2019.01)
CPC G06F 16/211 (2019.01) [G06F 16/219 (2019.01); G06F 16/2452 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method of managing a data pipeline, the method being performed by a processor of a data processing system and comprising:
obtaining data from one or more data sources associated with the data pipeline, the data being provided to one or more downstream consumers associated with the data pipeline;
making a first determination regarding whether the data comprises anomalous data containing a change in a system of representation of information that is associated with non-anomalous data;
in a first instance of the first determination in which the data comprises the anomalous data:
obtaining a translation schema intended to remediate the change in the system of representation of information within the anomalous data;
making a second determination regarding whether the translation schema successfully remediates the change in the system of representation of information indicated by the anomalous data; and
in an instance of the second determination in which the translation schema successfully remediates the change in the system of representation of information indicated by the anomalous data:
performing an action set to implement the translation schema in the data pipeline by at least:
generating an application programming interface (API) translation layer for the data pipeline; and
inserting the API translation layer into the data pipeline to obtain an updated data pipeline, the updated data pipeline being caused to use the API translation layer to transform the anomalous data into corrected data using the translation schema, the corrected data being provided to the one or more downstream consumers instead of the anomalous data.