US 12,406,185 B1
System and method for pruning neural networks at initialization using iteratively conserving synaptic flow
Hidenori Tanaka, Sunnyvale, CA (US); Daniel Kunin, Stanford, CA (US); Daniel L. K. Yamins, Sunnyvale, CA (US); and Surya Ganguli, Stanford, CA (US)
Assigned to NTT Research, Inc., Sunnyvale, CA (US); and THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY, Stanford, CA (US)
Filed by NTT Research, Inc., Sunnyvale, CA (US); and THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY, Stanford, CA (US)
Filed on Jul. 15, 2021, as Appl. No. 17/377,188.
Claims priority of provisional application 63/052,317, filed on Jul. 15, 2020.
Int. Cl. G06N 3/082 (2023.01); G06N 3/04 (2023.01)
CPC G06N 3/082 (2013.01) [G06N 3/04 (2013.01)] 16 Claims
OG exemplary drawing
 
1. An apparatus, comprising:
a computer system having an application specific integrated circuit (ASIC) and memory and a plurality of lines of instructions executed by the ASIC to perform operations comprising:
receive an untrained initial neural network having one or more layers with an input layer, an output layer and one or more layers between the input layer and the output layer in which each layer has a plurality of neurons with each neuron having a parameter, a particular neuron in each layer having a synapse that connects to another particular neuron in a different layer of the initial neural network;
initialize a binary mask that prunes one or more neurons in the initial neural network; and
perform iterative synaptic flow pruning to generate a sparse trainable subnetwork with a predetermined level of compression and no layer collapse.