US 12,084,085 B1
Systems and methods for autonomous vehicle validation
James Andrew Bagnell, Pittsburgh, PA (US); Sanjiban Choudhury, Ithaca, NY (US); Michael Lee Phillips, Pittsburgh, PA (US); Arun Venkatraman, Mountain View, CA (US); and Xinyan Yan, El Paso, TX (US)
Assigned to AURORA OPERATIONS, INC., Pittsburgh, PA (US)
Filed by Aurora Operations, Inc., Pittsburgh, PA (US)
Filed on Apr. 28, 2023, as Appl. No. 18/309,494.
Claims priority of provisional application 63/491,644, filed on Mar. 22, 2023.
Int. Cl. B60W 60/00 (2020.01); B60W 40/09 (2012.01)
CPC B60W 60/0015 (2020.02) [B60W 40/09 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for validating an operational system under test (SUT) for operating an autonomous vehicle, the method comprising:
(a) obtaining log data descriptive of an exemplar action of an exemplar vehicle in an environment, the exemplar action occurring in an initial state of the environment;
(b) determining, using the operational system, a planned action for a simulated vehicle in the initial state of the environment;
(c) simulating:
(i) a response of the simulated vehicle to an actor performing an actor action subsequent to the simulated vehicle executing the planned action in the initial state of the environment to obtain an SUT state of the environment; and
(ii) a response of the simulated vehicle to the actor performing the actor action subsequent to the simulated vehicle executing the exemplar action in the initial state of the environment to obtain an exemplar state of the environment;
(d) determining a test score based on the SUT state of the environment and a reference score based on the exemplar state of the environment; and
(e) evaluating the operational system based on the test score and the reference score.