US 12,293,291 B2
Systems and methods for time series analysis using attention models
Andreas Spanias, Tempe, AZ (US); Huan Song, Tempe, AZ (US); Jayaraman J. Thiagarajan, Milpitas, CA (US); and Deepta Rajan, Bellevue, WA (US)
Assigned to Arizona Board of Regents on Behalf of Arizona State University, Scottsdale, AZ (US); and Lawrence Livermore National Security, LLC, Livermore, CA (US)
Filed by Andreas Spanias, Tempe, AZ (US); Huan Song, Tempe, AZ (US); Jayaraman J. Thiagarajan, Milpitas, CA (US); and Deepta Rajan, Bellevue, WA (US)
Filed on Jul. 5, 2023, as Appl. No. 18/347,088.
Application 18/347,088 is a division of application No. 16/748,985, filed on Jan. 22, 2020, granted, now 11,699,079, issued on Jul. 11, 2023.
Claims priority of provisional application 62/795,176, filed on Jan. 22, 2019.
Prior Publication US 2023/0342611 A1, Oct. 26, 2023
Int. Cl. G06N 3/082 (2023.01); G06F 18/213 (2023.01); G06F 18/24 (2023.01); G06N 3/04 (2023.01); G06N 3/048 (2023.01); G06N 3/08 (2023.01)
CPC G06N 3/082 (2013.01) [G06F 18/213 (2023.01); G06N 3/04 (2013.01); G06N 3/048 (2023.01); G06N 3/08 (2013.01); G06F 18/24 (2023.01); G06F 2218/00 (2023.01)] 12 Claims
OG exemplary drawing
 
1. A computer-based method for analyzing and modeling a multivariate time series data based on an attention computation, the method comprising:
capturing dependencies across different variables through input embedding;
mapping an order of a sample appearance to a randomized lookup table via positional encoding;
capturing dependencies within a single sequence through a plurality of self-attention mechanisms, each self-attention mechanism of the plurality of self-attention mechanisms capturing dependencies within a single sequence of each self-attention mechanism;
determining a range of dependency to consider for each position being analyzed within the single sequence of each self-attention mechanism;
obtaining a plurality of attention weightings to other positions within the single sequence through computation of an inner product, each of the plurality of attention weightings obtained within the single sequence of each self-attention mechanism;
utilizing the plurality of attention weightings to acquire a plurality of vector representations for a position;
masking the single sequence of each self-attention mechanism to enable causality;
employing a dense interpolation technique for encoding partial temporal ordering to obtain a single vector representation from the plurality of vector representations;
applying a linear layer to obtain logits from the single vector representation; and
applying a final prediction layer comprising a sigmoid layer.