US 12,073,189 B2
Learned evaluation model for grading quality of natural language generation outputs
Thibault Sellam, New York City, NY (US); Dipanjan Das, Jersey City, NJ (US); and Ankur Parikh, New York, NY (US)
Assigned to Google LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Jun. 2, 2023, as Appl. No. 18/205,018.
Application 18/205,018 is a continuation of application No. 17/112,285, filed on Dec. 4, 2020, granted, now 11,704,506.
Application 17/112,285 is a continuation in part of application No. 17/003,572, filed on Aug. 26, 2020, granted, now 11,551,002, issued on Jan. 10, 2023.
Prior Publication US 2023/0306209 A1, Sep. 28, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 40/58 (2020.01); G06F 18/21 (2023.01); G06F 18/214 (2023.01); G06F 40/30 (2020.01); G06F 40/51 (2020.01); G06N 3/08 (2023.01)
CPC G06F 40/58 (2020.01) [G06F 18/214 (2023.01); G06F 18/2178 (2023.01); G06F 40/30 (2020.01); G06F 40/51 (2020.01); G06N 3/08 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of training a neural network, comprising:
generating, by one or more processors of a processing system, for each given synthetic sentence pair of a plurality of synthetic sentence pairs comprising a first passage of text and a second passage of text:
a first training signal of a plurality of training signals based on whether the given synthetic sentence pair was generated using backtranslation; and
one or more second training signals of the plurality of training signals based on a model prediction regarding a likelihood that one of the first passage of text or the second passage of text of the given synthetic sentence pair could have been generated by backtranslating the other one of the first passage of text or the second passage of text of the given synthetic sentence pair;
generating, by the one or more processors, a plurality of graded sentence pairs, each graded sentence pair of the plurality of graded sentence pairs comprising an original passage of text and a modified passage of text and a grade generated by a trained neural network based on the original passage of text and the modified passage of text; and
training, by the one or more processors, a student network to predict, for each given graded sentence pair in the plurality of graded sentence pairs, the grade generated by the neural network.