US 12,481,874 B2
Distributed adversarial training for robust deep neural networks
Sijia Liu, Somerville, MA (US); Gaoyuan Zhang, Medford, MA (US); Pin-Yu Chen, White Plains, NY (US); Chuang Gan, Cambridge, MA (US); and Songtao Lu, White Plains, NY (US)
Assigned to International Business Machines Corporation, Armonk, NY (US)
Filed by International Business Machines Corporation, Armonk, NY (US)
Filed on Feb. 8, 2021, as Appl. No. 17/170,343.
Prior Publication US 2022/0261626 A1, Aug. 18, 2022
Int. Cl. G06N 3/08 (2023.01); G06N 3/04 (2023.01)
CPC G06N 3/08 (2013.01) [G06N 3/04 (2013.01)] 15 Claims
OG exemplary drawing
 
1. A method for distributed adversarial training of a deep neural network-based model by distributed computing machines M to avoid misclassification of inputs by the deep neural network and reduce a number of epochs required for training of the deep neural-network based model, the method comprising:
obtaining, by each of the distributed computing machines M, adversarial perturbation-modified training examples for samples in a local dataset D(i), the training examples included labeled or unlabeled data;
computing, by each of the distributed computing machines M, local gradients of a local cost function ƒi with respect to parameters θ of the deep neural network-based model stored and trained locally on each of the distributed computing machines M, the local gradients of the local cost functions calculated using the adversarial perturbation-modified training examples stored locally on each of the distributed computing machines M;
transmitting, from each of the distributed computing machines M, the calculated local gradients of the local cost function ƒi to a server which aggregates the local gradients of the local cost function ƒi calculated by each of the distributed computing machines M and transmits an aggregated gradient to the distributed computing machines M, the aggregated gradient calculated from the local gradients;
receiving, by each of the distributed computing machines M the aggregated gradient from the server;
updating, by each of the distributed computing machines M, the parameters θ of the deep neural network-based model stored at each of the distributed computing machines M based on the aggregated gradient received from the server to generate an updated neural-network based model; and
classifying an input by one or more of the distributed computing machines M using an updated neural-based model, the updated neural-network based model robustly trained against worst-case loss induced by adversarially perturbed training examples.