US 12,450,518 B2
Evaluating on-device machine learning model(s) based on performance measures of client device(s) and/or the on-device machine learning model(s)
Dragan Zivkovic, Sunnyvale, CA (US); Akash Agrawal, San Jose, CA (US); Françoise Beaufays, Mountain View, CA (US); and Tamar Lucassen, Campbell, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Mar. 29, 2021, as Appl. No. 17/215,588.
Prior Publication US 2022/0309389 A1, Sep. 29, 2022
Int. Cl. G06N 20/00 (2019.01); G06N 5/04 (2023.01)
CPC G06N 20/00 (2019.01) [G06N 5/04 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A method implemented by one or more processors of a client device, the method comprising:
causing the client device to generate a plurality of performance measures for a dormant machine learning (ML) model based on a plurality of testing instances for the dormant ML model,
wherein the client device has a first set of device characteristics,
wherein the first set of device characteristics includes at least a first set of hardware characteristics,
wherein the client device has on-device memory storing the dormant ML model and the plurality of testing instances for the dormant ML model, and
wherein causing the client device to generate the plurality of performance measures based on a given testing instance, of the plurality of testing instance, comprises:
causing the client device to process, using the dormant ML model, testing instance input for the given testing instance to generate output; and
causing the client device to generate the plurality of performance measures based on the processing of the testing instance input of the given testing instance;
determining, based on the plurality of performance measures, whether to activate the dormant ML model at the client device, that has the first set of device characteristics, as an active ML model; and
in response to determining to activate the dormant ML model at the client device:
causing the dormant ML model to be activated for use locally at the client device as the active ML model; and
causing one or more additional client devices, that also have the first set of device characteristics, to utilize a corresponding instance of the dormant ML model for respective use, locally at the one or more additional client devices, as a corresponding instance of the active ML model.