US 11,755,947 B1
Offline evaluation of machine learning models with noise reduction
Juan Carlos Hernandez Munuera, Seattle, WA (US); Rizwana Rizia, Bothell, WA (US); Varun Narayan Hegde, Seattle, WA (US); Miguel Angel Hernandez Orozco, Kirkland, WA (US); Arnab Sinha, Issaquah, WA (US); Ravi Khandelwal, Bellevue, WA (US); and Lei Shi, Seattle, WA (US)
Assigned to Amazon Technologies, Inc., Seattle, WA (US)
Filed by Amazon Technologies, Inc., Seattle, WA (US)
Filed on Dec. 11, 2019, as Appl. No. 16/711,212.
Int. Cl. G06N 7/00 (2023.01); G06N 20/00 (2019.01)
CPC G06N 20/00 (2019.01) [G06N 7/00 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system, comprising: one or more processors and one or more memories to store computer-executable instructions that, when executed, cause the one or more processors to:
store a request log representing a plurality of historical requests to an application programming interface (API) in a production environment;
generate a plurality of replay requests based at least in part on the plurality of historical requests in the request log, wherein at least one parameter value varies for individual ones of the plurality of replay requests;
provide the plurality of replay requests to a production machine learning model that has been used to serve requests in a production environment and an experimental machine learning model, wherein the production machine learning model and the experimental machine learning model produce a set of results based at least in part on the plurality of replay requests;
determine a reduced set of results for which the production machine learning model and the experimental machine learning model differ; and
perform an evaluation of an impact of the experimental machine learning model using the reduced set of results and not using another subset of the results for which the production machine learning model and the experimental machine learning model did not differ.