US 12,455,942 B2
Experiment management service
Owen Thomas, Seattle, WA (US); Kenneth O Henderson, Jr., Everett, WA (US); Sumit Thakur, Seattle, WA (US); Glenn Danthi, Seattle, WA (US); Hugh Payton Staub, Seattle, WA (US); Thomas Albert Faulhaber, Seattle, WA (US); and Vladimir Zhukov, Seattle, WA (US)
Assigned to Amazon Technologies, Inc., Seattle, WA (US)
Filed by Amazon Technologies, Inc., Seattle, WA (US)
Filed on Feb. 17, 2023, as Appl. No. 18/171,244.
Application 18/171,244 is a continuation of application No. 16/894,707, filed on Jun. 5, 2020, granted, now 11,586,847.
Claims priority of provisional application 62/940,800, filed on Nov. 26, 2019.
Prior Publication US 2023/0281276 A1, Sep. 7, 2023
Int. Cl. G06F 9/44 (2018.01); G06F 8/71 (2018.01); G06F 9/451 (2018.01); G06F 9/48 (2006.01); G06F 9/54 (2006.01); G06F 16/14 (2019.01); G06F 16/16 (2019.01); G06F 18/2113 (2023.01); G06F 18/214 (2023.01); G06N 20/00 (2019.01); G06N 20/20 (2019.01)
CPC G06F 18/2148 (2023.01) [G06F 8/71 (2013.01); G06F 9/451 (2018.02); G06F 9/485 (2013.01); G06F 9/54 (2013.01); G06F 16/144 (2019.01); G06F 16/164 (2019.01); G06F 18/2113 (2023.01); G06F 18/2155 (2023.01); G06N 20/00 (2019.01); G06N 20/20 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method, comprising:
automatically collecting, without user-specification of the collecting and by a common framework configured to store and compare metrics for a plurality of related machine learning experiments conducted at least in part using resources of a cloud computing environment including a plurality of different machine learning framework resources selected for different ones of the plurality of related machine learning experiments, metrics generated at the selected plurality of different machine learning framework resources, wherein the related machine learning experiments include at least a first experiment which comprises a plurality of runs of a machine learning model;
in response to a first set of input received via one or more programmatic interfaces of the cloud computing environment,
providing, by the common framework, an indication of the plurality of related machine learning experiments;
in response to a second set of input received via the one or more programmatic interfaces,
providing, by the common framework, respective indications of individual runs of the plurality of runs; and
in response to a third set of input received via the one or more programmatic interfaces,
providing, by the common framework, an indication of one or more artifacts associated with a particular run of the plurality of runs, wherein the one or more artifacts include (a) a representation of a version of the machine learning model used in the particular run and (b) one or more of the metrics associated with one or more steps of the particular run and automatically collected based on the selected machine learning framework resource for the machine learning model used in the particular run without user-specification of collection of the one or more metrics.