US 11,810,011 B2
Generation of predictive model metrics using intermediate operations
William Benton, Madison, WI (US); and Erik Erlandson, Phoenix, AZ (US)
Assigned to Red Hat, Inc., Raleigh, NC (US)
Filed by Red Hat, Inc., Raleigh, NC (US)
Filed on Jan. 2, 2018, as Appl. No. 15/860,499.
Prior Publication US 2019/0205782 A1, Jul. 4, 2019
Int. Cl. G06N 7/01 (2023.01); G06N 5/02 (2023.01); G06N 20/00 (2019.01); G06N 5/022 (2023.01)
CPC G06N 7/01 (2023.01) [G06N 5/022 (2013.01); G06N 20/00 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
identifying a model comprising a plurality of intermediate operations, wherein each intermediate operation of the plurality of intermediate operations produces an intermediate output value for use by the model to generate a predictive output value, wherein the intermediate output value is different from the predictive output value;
for each of the plurality of intermediate operations, generating, by a processing device, a set of instructions to provide the intermediate output values when the model is executed;
performing a first execution of the identified model, wherein the first execution refers to an execution of a training set on the model;
receiving a first intermediate output value associated with a first intermediate operation used to produce a first predictive output value during the first execution of the model, wherein the first intermediate operation implements a decision block, and wherein the first predictive output value represents a number of positive responses associated with the decision block;
performing a second execution of the identified model, wherein the second execution refers to an execution of the model after the model has been implemented;
receiving a second intermediate output value associated with the first intermediate operation used to produce a second predictive output value during the second execution of the model;
determining that a difference between the first intermediate output value and the second intermediate output value satisfies a threshold number of positive responses associated with the decision block; and
in response to determining that the difference between the first intermediate output value and the second intermediate output value satisfies the threshold, transmitting, by the processing device, a notification indicating data drift of the model in view of the difference between the first intermediate output value and the second intermediate output value satisfies the threshold.