US 12,346,404 B1
Decoder-only transformer model for time series data
Ruiwen Zhang, Cary, NC (US); Bingfeng (Ben) Ding, Cary, NC (US); Samuel Paul Leeman-Munk, Durham, NC (US); Rui Liu, Cary, NC (US); and Lochan Basnet, Morrisville, NC (US)
Assigned to SAS Institute Inc., Cary, NC (US)
Filed by SAS Institute Inc., Cary, NC (US)
Filed on Dec. 23, 2024, as Appl. No. 18/991,935.
Claims priority of provisional application 63/643,337, filed on May 6, 2024.
Claims priority of provisional application 63/635,105, filed on Apr. 17, 2024.
Int. Cl. G06F 17/16 (2006.01); G06F 5/01 (2006.01); G06F 17/15 (2006.01)
CPC G06F 17/16 (2013.01) [G06F 5/01 (2013.01); G06F 17/15 (2013.01)] 30 Claims
OG exemplary drawing
 
1. A non-transitory computer-readable medium comprising computer-readable instructions that when executed by a processor cause the processor to:
receive a long sequence time series data, the long sequence time series data comprising a plurality of data points, each data point of the plurality of data points associated with a time stamp; and
forecast a series of future data points in the long sequence time series data using a decoder-only transformer model by:
creating an embedding for the long sequence time series data in an embedding layer of the decoder-only transformer model by:
dividing the long sequence time series data into a plurality of sequences, each sequence of the plurality of sequences having consecutive n data points of the plurality of data points, wherein each sequence of the plurality of sequences is offset from a neighboring sequence of the plurality of sequences based on a shift window;
converting each sequence of the plurality of sequences into a first vector to obtain a plurality of first vectors;
creating a plurality of second vectors from the time stamps associated with the plurality of data points, wherein each second vector of the plurality of second vectors corresponds to one sequence of the plurality of sequences; and
combining the first vector with the second vector of each sequence of the plurality of sequences to obtain a plurality of third vectors, wherein the plurality of third vectors corresponds to the embedding;
computing a context matrix in a decoder layer of the decoder-only transformer model based on the embedding;
inputting the context matrix into a prediction layer of the decoder-only transformer model;
performing a convolution operation on the context matrix to forecast the series of future data points; and
outputting the series of future data points from the prediction layer.