US 11,922,303 B2
Systems and methods for distilled BERT-based training model for text classification
Wenhao Liu, Redwood City, CA (US); Ka Chun Au, Milbrae, CA (US); Shashank Harinath, San Francisco, CA (US); Bryan McCann, Menlo Park, CA (US); Govardana Sachithanandam Ramachandran, Palo Alto, CA (US); Alexis Roos, Los Angeles, CA (US); and Caiming Xiong, Menlo Park, CA (US)
Assigned to Salesforce, Inc., San Francisco, CA (US)
Filed by Salesforce.com, Inc., San Francisco, CA (US)
Filed on May 18, 2020, as Appl. No. 16/877,339.
Claims priority of provisional application 62/968,973, filed on Jan. 31, 2020.
Claims priority of provisional application 62/937,085, filed on Nov. 18, 2019.
Prior Publication US 2021/0150340 A1, May 20, 2021
Int. Cl. G06N 3/00 (2023.01); G06F 40/40 (2020.01); G06N 3/045 (2023.01); G06N 3/08 (2023.01)
CPC G06N 3/08 (2013.01) [G06F 40/40 (2020.01); G06N 3/045 (2023.01)] 20 Claims
OG exemplary drawing
 
1. A method for distilling knowledge from a first neural network to train a second neural network, the method comprising:
receiving a plurality of training samples corresponding to a first set of pre-defined classes from a given dataset;
retrieving the first neural network that is pre-trained to classify input samples into the first set of pre-defined classes;
obtaining a first plurality of classifications by feeding the plurality of training samples to the first neural network;
transforming, using an out-of-distribution (OOD) sample generation module, one or more of the plurality of the training samples into one or more out-of-distribution (OOD) training samples, wherein the transforming further includes:
computing, using a term frequency-inverse document frequency model, inter-class word importance probabilities of one or more words of a training sample in the plurality of training samples, wherein the inter-class word importance probabilities indicate that the one or more words distinguish between the first plurality of classifications;
computing, using a discriminator model, in-distribution word importance probabilities of the one or more words of a training sample in the plurality of training samples, wherein the in-distribution word importance probabilities indicate contributions of the one or more words to a classification in the first plurality of classifications;
identifying a set of the one or more words within the training sample based on the inter-class word importance probabilities and the in-distribution word importance probabilities; and
replacing the set of the one or more words within the training sample with one or more random words;
generating a second set of classes by adding an out-of-distribution class to the first set of pre-defined classes; and
training the second neural network defined with the second set of classes based on the plurality of training samples, the one or more out-of-distribution training samples and the first plurality of classifications from the first neural network.