US 12,236,342 B2
Tensor ring decomposition for neural networks
Zhiyu Cheng, Sunnyvale, CA (US)
Assigned to Baidu USA LLC, Sunnyvale, CA (US)
Filed by Baidu USA, LLC, Sunnyvale, CA (US)
Filed on Oct. 21, 2020, as Appl. No. 17/076,184.
Prior Publication US 2022/0121926 A1, Apr. 21, 2022
Int. Cl. G06N 3/08 (2023.01); G06N 3/04 (2023.01)
CPC G06N 3/08 (2013.01) [G06N 3/04 (2013.01)] 16 Claims
OG exemplary drawing
 
1. A computer-implemented method for reducing resource requirements for a neural model comprising:
given a trained multi-dimensional neural network comprising one or more neural network (NN) layers:
for each neural network layer from a set of one or more neural network layers from the trained multi-dimensional neural network, using tensor ring (TR) more neural network layers from the trained multi-dimensional neural network, using tensor ring (TR) decomposition to approximate the neural network layer to obtain a TR-decomposed NN layer and to obtain a corresponding TR-decomposed multi-dimensional neural network that comprises one or more TR-decomposed NN layers;
using validation data on the TR-decomposed NN layer or on the TR-decomposed multi-dimensional neural network to determine if an output from the TR-decomposed NN layer or the TR-decomposed multi-dimensional neural network is within an acceptable threshold as compared to an output from a corresponding NN layer from the trained multi-dimensional neural network or from the trained multi-dimensional neural network;
responsive to the output not meeting an acceptable threshold value:
increasing a rank for TR-decomposition of one or more of the NN layers in the set of one or more neural network layers; and
returning to the step of, for each neural network layer from a set of one or more neural network layers from the trained multi-dimensional neural network, using tensor ring (TR) decomposition to approximate the neural network layer to obtain a TR-decomposed NN layer and to obtain a corresponding TR-decomposed multi-dimensional neural network that comprises one or more TR-decomposed NN layers; and
responsive to the output meeting an acceptable threshold value:
re-training the TR-decomposed multi-dimensional neural network using a training dataset until a stop condition is reached; and
outputting the re-trained TR-decomposed multi-dimensional neural network;
wherein the trained multi-dimensional neural network is a trained three-dimensional convolutional neural network comprising one or more convolutional neural network layers and the method comprises:
given the trained three-dimensional convolutional neural network (3D CNN) comprising one or more convolutional neural network (CNN) layers:
for each CNN layer from a set of one or more CNN layers from the trained 3D CNN, using tensor ring (TR) decomposition to approximate the CNN layer to obtain a TR-decomposed CNN layer and to obtain a corresponding TR-decomposed 3D CNN that comprises one or more TR-decomposed CNN layers;
using validation data on the TR-decomposed CNN layer or on the TR-decomposed 3D CNN to determine if an output is within an acceptable threshold as compared to an output from a corresponding CNN layer from the trained 3D CNN or from the trained 3D CNN;
responsive to the output not meeting an acceptable threshold value:
increasing a rank for TR-decomposition of one or more of the CNN layers in the set of one or more CNN layers; and
returning to the step of, for each CNN layer from a set of one or more CNN layers from the trained 3D CNN, using tensor ring (TR) decomposition to approximate the CNN layer to obtain a TR-decomposed CNN layer and to obtain a corresponding TR-decomposed 3D CNN that comprises one or more TR-decomposed CNN layers; and
responsive to the output meeting an acceptable threshold value:
re-training the TR-decomposed 3D CNN using a training dataset until a stop condition is reached; and
outputting the re-trained TR-decomposed 3D CNN, which requires less memory to store than the trained 3D CNN.