US 12,327,085 B2
Sentence similarity scoring using neural network distillation
Oren Barkan, Tel-Aviv (IL); Noam Razin, Herzliya (IL); and Noam Koenigstein, Herzliya (IL)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on Jun. 20, 2022, as Appl. No. 17/844,680.
Application 17/844,680 is a continuation of application No. 16/789,385, filed on Feb. 12, 2020, granted, now 11,392,770.
Claims priority of provisional application 62/946,911, filed on Dec. 11, 2019.
Prior Publication US 2022/0318507 A1, Oct. 6, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 40/284 (2020.01); G06N 3/045 (2023.01); G06N 3/08 (2023.01)
CPC G06F 40/284 (2020.01) [G06N 3/045 (2023.01); G06N 3/08 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system for attentive sentence similarity scoring, the system comprising:
a computer-readable medium storing instructions that are operative upon execution by a processor to:
precompute a plurality of candidate sentences using a trained distilled sentence embedding (DSE) language model, wherein the trained DSE language model generates a plurality of candidate sentence representations based on the plurality of candidate sentences;
receive an input query including a selected sentence;
generate a set of similarity scores for each candidate sentence in the plurality of candidate sentences and the selected sentence by comparing a representation of the selected sentence to each candidate sentence representation in the plurality of candidate sentence representations, wherein the trained DSE language model is trained using training pairs of sentences and a set of teacher similarity scores obtained from a transformer language model, and wherein the trained DSE language model is trained to approximate the set of teacher similarity scores for the training pairs of sentences using a low-cost similarity function;
select a set of similar sentences from the plurality of candidate sentences based on the set of similarity scores; and
generate a response to the input query comprising the set of similar sentences.