US 12,332,760 B2
Consistency monitoring of data in a data pipeline
J. Mitchell Haile, Carlisle, MA (US)
Assigned to Data Culpa, Inc., Carlisle, MA (US)
Filed by Data Culpa, Inc., Carlisle, MA (US)
Filed on Jan. 19, 2024, as Appl. No. 18/417,469.
Application 18/417,469 is a continuation of application No. 17/874,991, filed on Jul. 27, 2022, granted, now 11,914,496.
Claims priority of provisional application 63/228,734, filed on Aug. 3, 2021.
Prior Publication US 2024/0232041 A1, Jul. 11, 2024
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 11/34 (2006.01); G06F 9/54 (2006.01); G06F 11/07 (2006.01)
CPC G06F 11/3409 (2013.01) [G06F 9/542 (2013.01); G06F 11/0772 (2013.01)] 14 Claims
OG exemplary drawing
 
1. A system to monitor data consistency in a data pipeline, the system comprising:
processing circuitry configured to:
monitor operations of the data pipeline;
utilize a Sigmoid function to calculate weighted sums of data values, data records, and data schemas in the pipeline outputs;
generate a consistency score that indicates a similarity level between pipeline outputs based on the weighted sums;
determine when the consistency score exceeds a threshold; and
when the consistency score exceeds the threshold, transfer an alert that indicates ones of the pipeline outputs that exceeded the threshold value.