	US 12,236,206 B1
	Pretraining a language machine-learning model
Michael William Lewis, Seattle, WA (US); Marjan Ghazvini Nejad, Seattle, WA (US); Gargi Ghosh, Bellevue, WA (US); Armen Aghajanyan, Bellevue, WA (US); Sida Wang, Bellevue, WA (US); and Luke Zettlemoyer, Seattle, WA (US)
Assigned to Meta Platforms, Inc., Menlo Park, CA (US)
Filed by Meta Platforms, Inc., Menlo Park, CA (US)
Filed on May 26, 2021, as Appl. No. 17/331,478.
Int. Cl. G06F 40/58 (2020.01); G06F 18/22 (2023.01); G06N 3/048 (2023.01); G06N 3/08 (2023.01)

CPC G06F 40/58 (2020.01) [G06F 18/22 (2023.01); G06N 3/048 (2023.01); G06N 3/08 (2013.01)]

20 Claims

1. A method comprising:

accessing a first document;

accessing a plurality of second documents;

determining, for the plurality of second documents, corresponding relevance scores indicating a degree of relevance of the plurality of second documents to the first document using a machine-learning model;

selecting a subset of the plurality of second documents based on the corresponding relevance scores;

generating a target document by using the machine-learning model to process the subset of the plurality of second documents and the corresponding relevance scores; and

updating parameters of the machine-learning model based on a comparison between the first document and the generated target document.