US 11,948,063 B2
Improving a deep neural network with node-to-node relationship regularization
James K. Baker, Maitland, FL (US); and Bradley J. Baker, Berwyn, PA (US)
Assigned to D5AI LLC, Maitland, FL (US)
Filed by D5AI LLC, Maitland, FL (US)
Filed on Jun. 1, 2023, as Appl. No. 18/327,527.
Application 18/327,527 is a continuation of application No. 17/387,211, filed on Jul. 28, 2021, granted, now 11,836,600.
Claims priority of provisional application 63/068,080, filed on Aug. 20, 2020.
Prior Publication US 2023/0385608 A1, Nov. 30, 2023
Int. Cl. G06N 3/045 (2023.01)
CPC G06N 3/045 (2023.01) 26 Claims
OG exemplary drawing
 
1. A method of improving a base neural network, the method comprising:
initial training, at least partially, with a computer system that comprises one or more processing cores, a base neural network through machine learning with a training data set, wherein:
the base neural network comprises an input layer, an output layer, and one or more hidden layers between the input layer and the output layer;
each layer comprises one or more base network nodes, such that the base neural network comprises multiple base network nodes; and
the initial training comprises storing, in a memory of the computer system, preliminary activations values computed for the base network nodes on each of the one or more hidden layers for data in the training data set;
after the initial training of the base neural network, merging, by the computer system, a new node set into the base neural network to form an expanded neural network, wherein:
the new node set comprises one or more nodes; and
merging the new node set and the base neural network comprises directly connecting each of the one or more nodes of the new node set to one or more base network nodes in the expanded neural network; and
after merging the new node set into the base neural network to form the expanded neural network, training, by the computer system, the expanded neural network through iterative machine learning on the training data set using a network error loss function for the expanded neural network, wherein training the expanded neural network comprises imposing a node-to-node relationship regularization for at least one base network node in the expanded neural network, wherein imposing the node-to-node relationship regularization comprises adding, during back-propagation of partial derivatives through the expanded neural network for a datum in the training data set, a regularization cost to the network error loss function for the at least one base network node based on a specified relationship between a stored preliminary activation value for the base network node for the datum and an activation value for the base network node of the expanded neural network for the datum, wherein the preliminary activation value for the base network node of the base neural network for the datum was stored in the memory.