US 11,669,558 B2
Encoder using machine-trained term frequency weighting factors that produces a dense embedding vector
Yan Wang, Mercer Island, WA (US); Ye Wu, Bothell, WA (US); Houdong Hu, Redmond, WA (US); Surendra Ulabala, Bothell, WA (US); Vishal Thakkar, Kirkland, WA (US); and Arun Sacheti, Sammamish, WA (US)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on Mar. 28, 2019, as Appl. No. 16/368,798.
Prior Publication US 2020/0311542 A1, Oct. 1, 2020
Int. Cl. G06N 3/04 (2023.01); G06N 5/02 (2023.01); G06N 3/045 (2023.01); G06F 16/33 (2019.01); G06F 16/245 (2019.01); G06F 16/248 (2019.01); G06V 20/62 (2022.01); G06F 18/2413 (2023.01); G06F 17/16 (2006.01)
CPC G06F 16/3347 (2019.01) [G06F 16/245 (2019.01); G06F 16/248 (2019.01); G06F 18/2413 (2023.01); G06N 3/04 (2013.01); G06N 3/045 (2023.01); G06N 5/02 (2013.01); G06V 20/62 (2022.01); G06F 17/16 (2013.01)] 20 Claims
OG exemplary drawing
 
1. One or more computing devices for processing an instance of text, comprising:
hardware logic circuitry, the hardware logic circuitry including: (a) one or more hardware processors that perform operations by executing machine-readable instructions stored in a memory, and/or (b) one or more other hardware logic units that perform operations using a task-specific collection of logic gates, the operations including:
receiving an instance of input text in response to an action taken using a user computing device;
generating an input term-frequency (TF) vector that includes frequency information relating to frequency of occurrence of terms in the input text, the input TF vector corresponding to a vector having a prescribed dimensionality and having a plurality of TF values, each dimension of the input TF vector corresponding to a term in a vocabulary of predetermined size;
using a TF-modifying neural network to modify the frequency information in the input TF vector, associated with respective terms in the vocabulary, by respective machine-trained weighting factors associated with respective dimensions of the input TF vector, to produce an intermediate vector having a same dimensionality as the input TF vector, the TF-modifying neural network operating by modifying each particular TF value in the input TF vector by a corresponding machine-trained weighting factor that is specifically associated with the particular TF value, the TF-modifying neural network being implemented by the hardware logic circuitry and including at least one layer of neurons;
using a projection neural network to project the intermediate vector into an embedding vector having a dimensionality that is less than the dimensionality of the input TF vector, the embedding vector providing a distributed compact representation of semantic information in the input text, the projection neural network being implemented by the hardware logic circuitry and including at least one layer of neurons; and
utilizing the embedding vector to produce an output result that is accessible by the user computing device.