US 12,217,156 B2
Computing temporal convolution networks in real time
Piyush Khandelwal, Austin, TX (US); James MacGlashan, Riverside, RI (US); Peter Wurman, Acton, MA (US); and Fabrizio Santini, Arlington, MA (US)
Assigned to SONY GROUP CORPORATION, Tokyo (JP)
Filed by Sony Corporation, Tokyo (JP); and Sony Corporation of America, New York, NY (US)
Filed on Aug. 20, 2020, as Appl. No. 16/998,761.
Claims priority of provisional application 63/003,803, filed on Apr. 1, 2020.
Prior Publication US 2021/0312258 A1, Oct. 7, 2021
Int. Cl. G06N 3/049 (2023.01); G06N 3/08 (2023.01)
CPC G06N 3/049 (2013.01) [G06N 3/08 (2013.01)] 11 Claims
OG exemplary drawing
 
1. A computerized method of performing deep convolutions over time series data in real time in a temporal convolutional network (TCN), comprising:
training the temporal convolution network via a pruned TCN derived from a fixed-window view of the TCN, wherein the temporal convolution network is trained using fixed time splices of an input time series, wherein the fixed time splices have a size to provide a single output, and wherein unused convolutions of the fixed-window view are removed to generate the pruned TCN;
calculating network strides and dilation rates for each layer in a continual view of the TCN, where the calculated network strides and the calculated dilation rates are used in real time; and
executing the fixed-window view trained temporal convolution network in real time by:
running an algorithm once to generate buffers to store input and intermediate data, the algorithm determining a dilated kernel size each layer of the TCN, which is then used to initialize a single fixed size buffer in channels-width layout wherein a width in the channels-width layout references a time dimension, the algorithm further initializing a write head which identifies a column along the width where a next input to a buffer should be written;
inputting a first fixed time window of time series data;
predicting a first output through the trained temporal convolution network;
inputting a second fixed time window of time series data; and
predicting a second output through the trained temporal convolution network, wherein
a convolution operation that is used to provide the first output and is needed for the second output is not repeated for the prediction of the second output or for a later prediction output; and
a ready convolution operation is performed at a given timestep as soon as all inputs necessary to perform the ready convolution operation are available, even when the ready convolution operation is not used for generating an output at the given timestep.