US 12,454,280 B2
Metrics for evaluating autonomous vehicle performance
Skanda Shridhar, Pittsburgh, PA (US); Yuhang Ma, Pittsburgh, PA (US); Tara Lynn Stentz, Pittsburgh, PA (US); Zhengdi Shen, Pittsburgh, PA (US); Galen Clark Haynes, Pittsburgh, PA (US); and Neil Traft, Burlington, VT (US)
Assigned to Aurora Operations, Inc., Pittsburgh, PA (US)
Filed by Aurora Operations, Inc., Pittsburgh, PA (US)
Filed on Oct. 1, 2021, as Appl. No. 17/492,142.
Claims priority of provisional application 63/086,306, filed on Oct. 1, 2020.
Prior Publication US 2022/0105955 A1, Apr. 7, 2022
Int. Cl. B60W 60/00 (2020.01); B60W 40/08 (2012.01); B60W 40/10 (2012.01); G06N 20/00 (2019.01)
CPC B60W 60/001 (2020.02) [B60W 40/08 (2013.01); B60W 40/10 (2013.01); G06N 20/00 (2019.01); B60W 2040/0881 (2013.01); B60W 2540/01 (2020.02); B60W 2554/4049 (2020.02)] 20 Claims
OG exemplary drawing
 
1. An autonomous vehicle computing system for controlling an autonomous vehicle, the autonomous vehicle computing system comprising:
one or more processors;
a machine-learned prediction system having one or more parameters configured by:
(a) receiving data indicative of a trajectory of a test autonomous vehicle comprising a plurality of footprints of the test autonomous vehicle at a corresponding plurality of time steps of a time frame;
(b) receiving testing object data indicative of a ground truth object trajectory of a test object in a test environment of the test autonomous vehicle;
(c) receiving, from the machine-learned prediction system, prediction data indicative of one or more predicted object trajectories in the test environment;
(d) evaluating the machine-learned prediction system using a performance metric that characterizes protection of exposed ground truth occupancy, wherein the performance metric is based at least in part on, for a respective footprint of the plurality of footprints of the test autonomous vehicle:
(i) a probability that the ground truth object trajectory occupies at least a portion of the respective footprint at a respective time step corresponding to the respective footprint; and
(ii) a probability that the respective footprint is not blocked by at least one of the one or more predicted object trajectories before the respective time step; and
(e) updating, based at least in part on the evaluating of the machine-learned prediction system using the performance metric, the machine-learned prediction system to improve a value of the performance metric; and
one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle computing system to perform operations, the operations comprising:
processing sensor data descriptive of an object in an environment of the autonomous vehicle;
generating, using the machine-learned prediction system, one or more predictions for the object; and
controlling the autonomous vehicle based on the one or more predictions.