US 12,277,391 B2
Cross-attention between sparse external features and contextual word embeddings to improve text classification
Jean-Michel Attendu, Montréal (CA); Alexandre Jules Dos Santos, Montréal (CA); and François Duplessis Beaulieu, Montréal (CA)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by NUANCE COMMUNICATIONS, INC., Burlington, MA (US)
Filed on Jun. 10, 2022, as Appl. No. 17/837,475.
Prior Publication US 2023/0401383 A1, Dec. 14, 2023
Int. Cl. G06F 40/279 (2020.01); G06F 40/284 (2020.01); G06F 40/30 (2020.01); G06N 3/04 (2023.01)
CPC G06F 40/284 (2020.01) 20 Claims
OG exemplary drawing
 
1. A method comprising:
obtaining a dense representation of external features, said dense representation of external features being obtained by multiplying a sparse representation of said external features with a first dense matrix;
obtaining, by a natural language understanding (NLU) application, a dense representation of text, said dense representation of text being obtained by multiplying a sparse representation of said text with a second dense matrix;
obtaining a mask that associates said external features to tokens of said text, the mask being a sparse matrix that represents which external feature is associated with which token of said text;
performing, by a cross-attention process that utilizes said mask, an information fusion of said dense representation of said external features and said tokens of said text;
based on said information fusion, generating, by the NLU application, a joint representation of said external features and said tokens of said text; and
performing, by the NLU application, named entity recognition using the joint representation.