US 12,229,219 B2
Dynamic evaluation of model acceptability
Hui Wang, Beijing (CN); Zhao Yu Wang, Beijing (CN); Jing Wen Chen, Beijing (CN); Wei Song, Beijing (CN); Li Cao, Bei Jing (CN); Wan Yue Chen, Beijing (CN); and Wen Zhong Liu, Beijing (CN)
Assigned to INTERNATIONAL BUSINESS MACHINES CORPORATION, Armonk, NY (US)
Filed by International Business Machines Corporation, Armonk, NY (US)
Filed on Feb. 3, 2021, as Appl. No. 17/165,993.
Prior Publication US 2022/0245393 A1, Aug. 4, 2022
Int. Cl. G06F 18/22 (2023.01); G06F 17/18 (2006.01); G06F 18/25 (2023.01); G06V 10/75 (2022.01); G06N 20/00 (2019.01)
CPC G06F 18/22 (2023.01) [G06F 17/18 (2013.01); G06F 18/25 (2023.01); G06V 10/751 (2022.01); G06N 20/00 (2019.01)] 12 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
training a neural network to predict a value of real-time data prior to the real-time data being available;
generating, with the trained neural network and responsive to receiving current real-time data, a plurality of predictions, wherein each prediction is a prediction for a future real-time input to a model;
collecting, after generating the plurality of predictions, actual values for the future real-time inputs to the model;
generating a deviation curve encoding, in real-time, a distance between each prediction of a future real-time input to the model and the respective collected actual value for the future real-time input to the model;
sampling one or more points in the deviation curve, wherein each sampled point comprises the distance between the associated prediction and the respective collected actual value for the future real-time input;
determining a sample point gradient in the deviation curve for each sampled point;
passing the sampled points and the respective sample point gradients to a compare module, the compare module configured for:
comparing the sampled points to a low threshold and a high threshold; and
at each time step, accepting or denying the respective prediction as an input to the model based on the comparison of the respective sampled point to the low threshold and the high threshold and a value of the respective sample point gradient, and wherein the decision associated with a respective sampled point is overridden responsive to a value of the respective sample point gradient being above a predetermined threshold; and
modifying the plurality of future inputs for the model by selectively replacing a subset of the predictions with their corresponding collected actual values for the future real-time inputs according to the acceptance or denial at each time step.