US 12,118,064 B2
Training machine learning models using unsupervised data augmentation
Thang Minh Luong, Mountain View, CA (US); Quoc V. Le, Sunnyvale, CA (US); Qizhe Xie, Pittsburgh, PA (US); and Zihang Dai, Pittsburgh, PA (US)
Assigned to Google LLC, Mountain View, CA (US)
Appl. No. 17/606,190
Filed by Google LLC, Mountain View, CA (US)
PCT Filed Apr. 24, 2020, PCT No. PCT/US2020/029945
§ 371(c)(1), (2) Date Oct. 25, 2021,
PCT Pub. No. WO2020/219971, PCT Pub. Date Oct. 29, 2020.
Claims priority of provisional application 62/838,932, filed on Apr. 25, 2019.
Prior Publication US 2022/0215209 A1, Jul. 7, 2022
Int. Cl. G06F 18/21 (2023.01); G06F 18/214 (2023.01); G06N 3/08 (2023.01)
CPC G06F 18/217 (2023.01) [G06F 18/2148 (2023.01); G06N 3/08 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
receiving training data for training a machine learning model to map model inputs to model outputs in order to perform a particular machine learning task, the training data comprising:
a plurality of unlabeled training inputs; and
a plurality of labeled training inputs and, for each labeled training input, a ground truth output that should be generated by the machine learning model by performing the particular machine learning task on the labeled training input;
generating augmented training data, comprising generating, for each of the plurality of unlabeled training inputs, a respective augmented training input by applying a data augmentation technique to the unlabeled training input;
training the machine learning model on the augmented training data, comprising:
training the machine learning model on the unlabeled training inputs and the augmented training inputs to optimize an unsupervised objective that measures a difference between (i) a model output generated by the machine learning model for a given unlabeled training input and (ii) a model output generated by the machine learning model for the augmented training input generated from the unlabeled training input, and
training the machine learning model on the labeled training inputs to optimize a supervised objective that measures a difference between (i) a model output generated by the machine learning model for a given labeled training input and (ii) the ground truth output for the given labeled training input.