| CPC G06N 5/04 (2013.01) [G06N 20/00 (2019.01)] | 20 Claims |
|
15. A system, comprising:
a computing device; and
a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for regression detection in machine learning (ML) models resulting from version changes in an application leveraging the ML models, the operations comprising:
receiving a project structure representing a regression test file directory for regression inference and comprising a set of test scenarios, the test file directory comprising a ML model file indicating one or more ML models to be tested for regression, one or more input files storing input data to be input to the one or more ML models during regression inference, an expected results file, and a test results file;
determining that a test scenario of the set of test scenarios is to be executed by validating the test scenario in view of the test file directory;
executing a test inference job using a second version of the application by processing input data of the one or more input files through the one or more ML models to provide a set of actual results comprising output of the one or more ML models, the test inference job representing the test scenario and comprising matching, by each of the ML models, target items and query items defined in the input data to generate the set of actual results;
receiving the set of actual results of the test inference job;
calculating a prediction score based on the set of actual results and a set of expected results of the test scenario, the set of expected results being defined in the expected results file and comprising an overall score of the ML models for a first version of the application; and
selectively indicating regression of the one or more ML models of the test scenario based on the prediction score and the overall score.
|