US 12,088,621 B2
Meta-learning and auto-labeling for machine learning
Nitin S. Sharma, San Jose, CA (US)
Assigned to PayPal, Inc., San Jose, CA (US)
Filed by PayPal, Inc., San Jose, CA (US)
Filed on Dec. 15, 2020, as Appl. No. 17/122,683.
Claims priority of provisional application 63/080,522, filed on Sep. 18, 2020.
Prior Publication US 2022/0092472 A1, Mar. 24, 2022
Int. Cl. H04L 9/40 (2022.01); G06F 18/21 (2023.01); G06F 18/213 (2023.01); G06F 18/214 (2023.01); G06F 18/2415 (2023.01); G06N 5/04 (2023.01); G06N 20/00 (2019.01)
CPC H04L 63/1441 (2013.01) [G06F 18/213 (2023.01); G06F 18/2148 (2023.01); G06F 18/217 (2023.01); G06F 18/2415 (2023.01); G06N 5/04 (2013.01); G06N 20/00 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method, comprising:
comparing, by a computer system, output of a new machine learning model for a new set of examples with known labels for examples in the new set of examples, wherein the new set of examples includes one or more new features;
determining, by the computer system based on the comparing, whether a current performance of the new machine learning model satisfies a performance threshold for machine learning models, wherein the performance threshold is based on output of a benchmark machine learning model and a distribution summary of features that are included in a set of testing examples used to test the performance of the benchmark machine learning model, and wherein the distribution summary is generated using a generative machine learning model; and
in response to determining that the current performance of the new model does not satisfy the performance threshold, automatically triggering, by the computer system, retraining of the new model using the one or more new features.