US 11,790,227 B1
Systems and methods for neural content scoring
Brian W. Riordan, Pennington, NJ (US); Kenneth Steimel, Bloomington, IN (US); Michael Flor, Lawrence Township, NJ (US); and Robert A. Pugh, San Jose, CA (US)
Assigned to Educational Testing Service, Princeton, NJ (US)
Filed by Educational Testing Service, Princeton, NJ (US)
Filed on Jan. 14, 2021, as Appl. No. 17/148,742.
Claims priority of provisional application 63/014,733, filed on Apr. 24, 2020.
Claims priority of provisional application 62/961,983, filed on Jan. 16, 2020.
Int. Cl. G06N 3/08 (2023.01); G06F 40/284 (2020.01); G06F 40/232 (2020.01)
CPC G06N 3/08 (2013.01) [G06F 40/232 (2020.01); G06F 40/284 (2020.01)] 20 Claims
OG exemplary drawing
 
1. A method of automatically scoring a constructed response using a neural network, comprising:
training the network from a set of examples with a set of mathematical operations to average network weights across training steps by exponential moving average or stochastic weight average;
receiving the constructed response at a processing system;
processing the constructed response with the processing system to divide the constructed response into multiple series of word tokens, wherein each word token includes a sequence of characters;
processing the constructed response with the processing system to correct one or more spelling errors by converting one or more of the word tokens into a canonical form;
encoding, with the processing system, the sequence of characters in each word token using character-level encoding to generate character representation vectors;
encoding, with the processing system, each word token using word-level encoding to generate word representation vectors;
concatenating the word and character representation vectors for each series of word tokens to generate a plurality of concatenated representation vectors for the constructed response;
applying a set of nonlinear operations to the plurality of concatenated representation vectors using a recurrent neural network to generate a single vector output, wherein the set of nonlinear operations comprise:
zt=σ(W(z)xt+U(z)ht−1+b(z));
rt=σ(W(r)xt+U(r)ht−1+b(r));
ht=tanh(W(z)xt+U(z)ht−1+b(z)); and
ht=zt∘ht−1+(1−zt)∘ht,
wherein each time step t has an input xt and hidden state ht; W(z), W(r), W ∈ custom characternH×nI; U(z), U(r), U ∈ custom characternH×nH; n is a hyperparameter; and
applying a set of predetermined network weights to the vector output of the recurrent neural network to generate a scalar output for scoring the constructed response.