US 11,842,045 B2
Modality learning on mobile devices
Yu Ouyang, San Jose, CA (US); Diego Melendo Casado, San Francisco, CA (US); Mohammadinamul Hasan Sheik, San Jose, CA (US); Francoise Beaufays, Mountain View, CA (US); Dragan Zivkovic, Sunnyvale, CA (US); and Meltem Oktem, Toronto (CA)
Assigned to Google LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Aug. 31, 2022, as Appl. No. 17/823,545.
Application 17/823,545 is a continuation of application No. 17/064,173, filed on Oct. 6, 2020, granted, now 11,435,898, issued on Sep. 6, 2022.
Application 17/064,173 is a continuation of application No. 15/393,676, filed on Dec. 29, 2016, granted, now 10,831,366, issued on Nov. 10, 2020.
Prior Publication US 2022/0413696 A1, Dec. 29, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 3/04886 (2022.01); G06F 3/16 (2006.01); G06F 1/16 (2006.01); G06F 3/023 (2006.01); G06F 3/04883 (2022.01); G06F 40/166 (2020.01); G06F 40/289 (2020.01); G10L 15/22 (2006.01)
CPC G06F 3/04886 (2013.01) [G06F 1/1626 (2013.01); G06F 3/0233 (2013.01); G06F 3/04883 (2013.01); G06F 3/167 (2013.01); G06F 40/166 (2020.01); G06F 40/289 (2020.01); G06F 2203/0381 (2013.01); G10L 15/22 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method, comprising:
receiving, by a computing device, user input of a particular term via a first user input mode associated with the computing device, wherein the particular term is a textual version of the user input;
determining, at the computing device, that a first modality recognition model corresponding to the first user input mode does not recognize the particular term;
responsive to the determining that the first modality recognition model does not recognize the particular term, displaying, by the graphical user interface, one or more candidate replacement terms to replace the particular term;
detecting, by the graphical user interface and in response to the displaying of the one or more candidate replacement terms, a user indication that the textual version is a correct version of the user input;
based on the user indication, adding the particular term to the first modality recognition model;
sending the particular term from the first modality recognition model to a cross-modality learning system, wherein the cross-modality learning system is configured to aggregate a plurality of parameter values based on parameter signals received from a plurality of modality recognition models, and to share the particular term and the aggregated parameter values with a second modality recognition model corresponding to a second user input mode associated with the computing device for use in cross-modality learning between the first modality recognition model and the second modality recognition model; and
updating, by the computing device and based on the particular term and the aggregated parameter values, the cross-modality learning for the second modality recognition model, wherein the updating causes the second modality recognition model to automatically recognize a subsequent user input via the second user input mode.