US 12,293,159 B1
Grammar practice item creation using language models
Sophia Chan, Coquitlam (CA); and Swapna Somasundaran, Plainsboro, NJ (US)
Assigned to Educational Testing Service, Princeton, NJ (US)
Filed by Educational Testing Service, Princeton, NJ (US)
Filed on May 17, 2022, as Appl. No. 17/746,194.
Claims priority of provisional application 63/189,796, filed on May 18, 2021.
Int. Cl. G06F 40/40 (2020.01); G06F 40/205 (2020.01); G06F 40/253 (2020.01)
CPC G06F 40/40 (2020.01) [G06F 40/205 (2020.01); G06F 40/253 (2020.01)] 19 Claims
OG exemplary drawing
 
1. A computer-implemented method for automatic content generation comprising:
receiving data comprising an original sentence with a grammar artifact of interest;
inputting the received data into a distractor generation function to generate a plurality of distractor candidates based on the original sentence with the grammar artifact of interest, each distractor candidate comprising a different variation of the original sentence with an error;
scoring, using at least one machine learning-based language model, each of the distractor candidates, the scores characterizing a likelihood of such distractor candidate being selected as part of an assessment by a subject, the scoring being based on: masking, for each of the distractor candidates, a location of each error within the corresponding variation of the original sentence; and obtaining a predicted token for the masked error locations from the machine learning-based language model;
filtering the distractor candidates to result in a filtered list of distractor candidates;
selecting x top scoring distractor candidates from the filtered list of distractor candidates;
generating a grammar practice item based on the original sentence and the x top scoring distractor candidates; and
providing the grammar practice item to a consuming application or process.