US 11,809,968 B2
Control of hyperparameter tuning based on machine learning
Austin Grant Walters, Savoy, IL (US); Jeremy Edward Goodsitt, Champaign, IL (US); Anh Truong, Champaign, IL (US); and Mark Louis Watson, Sedona, AZ (US)
Assigned to Capital One Services, LLC, McLean, VA (US)
Filed by Capital One Services, LLC, McLean, VA (US)
Filed on Feb. 24, 2020, as Appl. No. 16/799,227.
Prior Publication US 2021/0264199 A1, Aug. 26, 2021
Int. Cl. G06N 20/00 (2019.01); G06F 18/21 (2023.01); G06F 18/2115 (2023.01)
CPC G06N 20/00 (2019.01) [G06F 18/217 (2023.01); G06F 18/2115 (2023.01)] 20 Claims
OG exemplary drawing
 
1. A non-transitory computer-readable medium storing instructions configured to cause one or more processors to:
determine a plurality of batches of hyperparameters for an artificial intelligence (AI) model, the plurality of batches of hyperparameters configured to be tuned according to a hyperparameter tuning technique based on a success metric, each of the plurality of batches of hyperparameters comprising a different set of hyperparameters, the different set of hyperparameters for at least a portion of the plurality of batches of hyperparameters selected based on previous test results;
instantiate a batch of instances of the AI model, each of the instances of the AI model generated based on a different one the plurality of batches of hyperparameters;
for each instance of the AI model and a set of hyperparameters associated with the instance of the AI model, at least partially in parallel:
train the instance of the AI model using training data;
test the instance of the AI model using testing data to generate AI model testing results;
train a prediction model using a machine learning process and the AI model testing results, the prediction model configured to estimate whether further application of the hyperparameter tuning technique will cause an improvement in the success metric;
test the hyperparameters using the hyperparameter tuning technique until a stopping point;
apply the prediction model to determine if further testing of the hyperparameters after the stopping point is predicted to improve the success metric; and
terminate the hyperparameter tuning technique when:
(i) an accuracy of the prediction model in predicting improvement in the success metric is above a predetermined accuracy threshold, and the prediction model predicts that further application of the hyperparameter tuning technique will not result in an improvement to the success metric; or
(ii) the accuracy of the prediction model in predicting improvement in the success metric is below the predetermined accuracy threshold, and an accuracy of hyperparameter optimization is determined to be below a predetermined tuning accuracy threshold.