US 12,314,853 B2
Training agent trajectory prediction neural networks using distillation
Bertrand Robert Douillard, San Francisco, CA (US); and Dijia Su, Jersey City, NJ (US)
Assigned to Waymo LLC, Mountain View, CA (US)
Filed by Waymo LLC, Mountain View, CA (US)
Filed on Sep. 16, 2022, as Appl. No. 17/947,052.
Claims priority of provisional application 63/248,950, filed on Sep. 27, 2021.
Claims priority of provisional application 63/245,173, filed on Sep. 16, 2021.
Prior Publication US 2023/0082079 A1, Mar. 16, 2023
Int. Cl. G06K 9/62 (2022.01); G06N 3/08 (2023.01); G06T 7/20 (2017.01)
CPC G06N 3/08 (2013.01) [G06T 7/20 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30241 (2013.01)] 20 Claims
OG exemplary drawing
 
12. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a scene-centric trajectory prediction neural network that has a plurality of scene-centric parameters and that is configured to receive a scene input comprising features of a scene in an environment that includes a plurality of agents and to process the features in accordance with the scene-centric parameters to generate as output a respective trajectory prediction for each of the plurality of agents, the operations comprising:
obtaining a batch of one or more training examples, each training example comprising data characterizing a respective scene in the environment that includes at least a respective plurality of agents;
for each training example in the batch:
generating, from the training example, a respective agent-centric input for each of the respective plurality of agents in the respective scene characterized by the training example, the respective agent-centric input for each agent comprising features that characterize the respective scene and that are represented in an agent-specific coordinate frame that is specific to the agent;
for each agent in the respective scene, processing the agent-centric input using a trained agent-centric trajectory prediction neural network, wherein the trained agent-centric trajectory prediction neural network has been trained to receive the agent-centric input that comprises features that are represented in the agent-specific coordinate frame for the agent to generate a trajectory prediction for the agent;
generating, from the training example, a scene input characterizing the respective scene characterized by the training example in a shared coordinate frame that is common to all of the agents in the respective scene; and
processing the scene input using the scene-centric trajectory prediction neural network and in accordance with current values of the scene-centric parameters to generate as output a respective trajectory prediction for each of the respective plurality of agents in the respective scene;
determining a gradient with respect to the scene-centric parameters of a loss function that includes one or more terms that measure, for each training example and for each of the respective plurality of agents in the respective scene characterized by the training example, a difference between (i) the trajectory prediction generated for the agent by the trained agent-centric trajectory prediction neural network and (ii) the trajectory prediction generated for the agent by the scene-centric trajectory prediction neural network; and
updating, using the gradient, the current values of the scene-centric parameters of the scene-centric trajectory prediction neural network.