US 12,066,918 B2
System to track and measure machine learning model efficacy
Lokesh Nyati, Pune (IN); Jonathan Doering, London (GB); Sruthi Yapalapalli, Hyderabad (IN); and Sriharsha Vogeti, Hyderabad (IN)
Assigned to PAYPAL, INC., San Jose, CA (US)
Filed by PAYPAL, INC., San Jose, CA (US)
Filed on Mar. 6, 2023, as Appl. No. 18/118,056.
Application 18/118,056 is a continuation of application No. 17/024,133, filed on Sep. 17, 2020, granted, now 11,609,838.
Prior Publication US 2023/0205663 A1, Jun. 29, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 11/34 (2006.01); G06F 11/30 (2006.01); G06F 11/32 (2006.01); G06N 20/00 (2019.01)
CPC G06F 11/3452 (2013.01) [G06F 11/302 (2013.01); G06F 11/323 (2013.01); G06N 20/00 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A system, comprising:
a processor; and
a non-transitory computer-readable medium having instructions stored thereon that are executable by the processor to cause the system to perform operations comprising:
receiving via online monitoring by the system, a machine learning model performance information, wherein the performance information includes a first set of data fields produced as outputs to a deployed artificial intelligence algorithm and a second set of data fields collected as inputs to the deployed artificial intelligence algorithm;
calculating by the system, a first set of statistical metrics based on the first set of data fields and a second set of statistical metrics based on the second set of data fields;
comparing the second set of statistical metrics with a baseline statistical metric characterizing training data associated with the machine learning model; and
generating an alert based in part of the comparing, wherein the alert indicates an additional training of the machine learning model is recommended.