US 11,748,567 B2
Total correlation variational autoencoder strengthened with attentions for segmenting syntax and semantics
Dingcheng Li, Sammamish, WA (US); Shaogang Ren, Redmond, WA (US); and Ping Li, Bellevue, WA (US)
Assigned to Baidu USA LLC, Sunnyvale, CA (US)
Filed by Baidu USA, LLC, Sunnyvale, CA (US)
Filed on Jul. 10, 2020, as Appl. No. 16/926,525.
Prior Publication US 2022/0012425 A1, Jan. 13, 2022
Int. Cl. G06F 40/10 (2020.01); G06F 40/284 (2020.01); G06F 40/211 (2020.01); G06F 40/30 (2020.01); G06N 3/08 (2023.01)
CPC G06F 40/284 (2020.01) [G06F 40/211 (2020.01); G06F 40/30 (2020.01); G06N 3/08 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method for segmenting latent representations comprising:
given a variational autoencoder (VAE) model comprising an attention layer, a semantic encoder, a syntax encoder, and a decoder:
generating, using an embedding layer, a sequence of embeddings for a sequence of tokens, in which the tokens are words or representations of words;
generating, using the attention layer, a sequence of attention masks based on the sequence of embeddings;
generating a sequence of hidden variables based on the sequence of embeddings and the sequence of attention masks;
generating, using the semantic encoder, a first sequence of latent variables based on the sequence of hidden variables;
generating, using the syntax encoder, a second sequence of latent variables based on the sequence of hidden variables;
inferring, using the decoder, a sequence of reconstructed tokens and a sequence of reconstructed attention masks using at least information of the first sequence of latent variables from the semantic encoder and the second sequence of latent variables from the syntax encoder; and
using the sequence of reconstructed tokens and the sequence of reconstructed attention masks to train at least the attention layer, the semantic encoder, the syntax encoder, and the decoder of the VAE model.