US 11,989,626 B2
Generating performance predictions with uncertainty intervals
Matthew Richard Arnold, Ridgefield Park, NJ (US); Benjamin Tyler Elder, White Plains, NY (US); Jiri Navratil, Cortlandt Manor, NY (US); and Ganesh Venkataraman, White Plains, NY (US)
Assigned to International Business Machines Corporation, Armonk, NY (US)
Filed by INTERNATIONAL BUSINESS MACHINES CORPORATION, Armonk, NY (US)
Filed on Apr. 7, 2020, as Appl. No. 16/842,295.
Prior Publication US 2021/0312323 A1, Oct. 7, 2021
Int. Cl. G06N 20/00 (2019.01); G06F 11/34 (2006.01); G06F 17/18 (2006.01); G06N 20/20 (2019.01)
CPC G06N 20/00 (2019.01) [G06F 11/3495 (2013.01); G06F 17/18 (2013.01); G06N 20/20 (2019.01)] 25 Claims
OG exemplary drawing
 
1. A computer-implemented method, comprising:
obtaining a first machine learning model configured to perform a task and a first dataset comprising unlabeled data;
training a second machine learning model to predict a performance of the first machine learning model at performing the task on the first dataset, based at least in part on the first machine learning model and a second dataset comprising labeled data, wherein the second machine learning model is a meta-model associated with the first machine learning model;
after the first machine learning model has been deployed to perform the task, generating, using the second machine learning model, at least one metric predicting the performance of the first machine learning model at performing the task on the production first dataset;
generating, for each third machine learning model of a plurality of third machine learning models, a respective set of features for training the third machine learning model based on an evaluation of one or more third datasets, wherein each of the plurality of third machine learning models is a respective meta-meta-model associated with the second machine learning model and is configured to predict one or more uncertainties of the at least one metric predicting the performance of the first machine learning model at performing the task on the first dataset;
training each third machine learning model of the plurality of third machine learning models, based on the respective set of features corresponding to the third machine learning model;
dynamically selecting one of the plurality of third machine learning models, based on an evaluation of the first dataset, the one or more third datasets, and the sets of features;
after the first machine learning model has been deployed to perform the task, generating, using the selected one of the plurality of third machine learning models, at least one value predicting the one or more uncertainties of the at least one metric predicting the performance of the first machine learning model at performing the task on the first dataset; and
providing an indication of the at least one metric and the at least one value.