US 12,340,285 B2
Testing models in data pipeline
Sattwati Kundu, Bangalore (IN); and Samiulla Zakir Hussain Shaikh, Bangalore (IN)
Assigned to International Business Machines Corporation, Armonk, NY (US)
Filed by International Business Machines Corporation, Armonk, NY (US)
Filed on Sep. 1, 2021, as Appl. No. 17/463,673.
Prior Publication US 2023/0068513 A1, Mar. 2, 2023
Int. Cl. G06N 20/00 (2019.01); G06N 5/02 (2023.01)
CPC G06N 20/00 (2019.01) [G06N 5/02 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
in response to receiving a request to validate data for a new artificial intelligence (AI) model within an existing machine learning data pipeline, checking metadata of the new AI model for data requirements, model specific requirements, and data quality metrics;
generating a data profile for the new AI model, the data profile including metadata associated with the AI model, the metadata including data quality metrics, training data, test data, and payload data;
generating modified test data by perturbing the metadata in the data profile, the perturbing operating to introduce random changes to seasonality data of the data quality metrics to build model variants from parameter space sampling;
validating the modified test data as being relevant to a context of the existing pipeline covering the parameter space of the new AI model;
automatically generating one or more test scenarios using the modified test data and varying hyperparameters of the new AI model and the model variants;
executing the generated test scenarios;
determining an accuracy value of performance of the new AI model based on results of the executed test scenarios; and
automatically inserting the new AI model into the existing data pipeline.