US 11,720,825 B2
Framework for multi-tenant data science experiments at-scale
Sarah Aerni, San Francisco, CA (US); Luke Sedney, San Mateo, CA (US); Kin Fai Kan, Sunnyvale, CA (US); and Till Christian Bergmann, San Mateo, CA (US)
Assigned to Salesforce, Inc., San Francisco, CA (US)
Filed by salesforce.com, inc., San Francisco, CA (US)
Filed on Jan. 31, 2019, as Appl. No. 16/263,927.
Prior Publication US 2020/0250587 A1, Aug. 6, 2020
Int. Cl. G06N 20/20 (2019.01); G06F 11/34 (2006.01); G06N 5/043 (2023.01)
CPC G06N 20/20 (2019.01) [G06F 11/3466 (2013.01); G06N 5/043 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of modifying an automated machine learning system, the automated machine learning system comprising a plurality of machine learning algorithms, including an initial machine learning algorithm, wherein the automated machine learning system generates at least an initial machine learning model of a plurality of machine learning models based on the plurality of machine learning algorithms in response to a request for a prediction based on a first tenant's data, the method comprising:
receiving, through an experimental interface, experimental modeling data reflecting at least one modification to at least one of the plurality of machine learning algorithms of the initial machine learning model;
modifying the initial machine learning algorithm of the plurality of machine learning algorithms using the at least one modification to generate an experimental machine learning algorithm;
generating an experimental machine learning model using the experimental machine learning algorithm;
determining, based on metadata associated with model data from the initial machine learning model, which portion of the first tenant's data is being used to train the initial machine learning model;
generating an indication of a performance of the experimental machine learning model with the experimental modeling data operating on the determined portion of the first tenant's data; and
comparing the indication of the performance of the experimental machine learning model with a performance of at least the initial machine learning model of the plurality of machine learning models operating on the determined portion of the first tenant's data,
wherein the experimental interface restricts a machine learning system operator from viewing any of the first tenant's data based on a permission parameter.