US 12,422,791 B2
Simulation-based training of an autonomous vehicle
Daniela Rus, Weston, MA (US); Sertac Karaman, Cambridge, MA (US); Igor Gilitschenski, Cambridge, MA (US); Alexander Amini, Cambridge, MA (US); Julia Moseyko, Orinda, CA (US); and Jacob Phillips, Boston, MA (US)
Assigned to Massachusetts Institute of Technology, Cambridge, MA (US)
Filed by Massachusetts Institute of Technology, Cambridge, MA (US)
Filed on Jun. 11, 2021, as Appl. No. 17/345,738.
Claims priority of provisional application 63/038,376, filed on Jun. 12, 2020.
Prior Publication US 2021/0389776 A1, Dec. 16, 2021
Int. Cl. G05B 13/04 (2006.01); G06F 16/50 (2019.01); G06T 3/00 (2024.01); G06T 7/50 (2017.01)
CPC G05B 13/04 (2013.01) [G06F 16/50 (2019.01); G06T 3/00 (2013.01); G06T 7/50 (2017.01); G06T 2207/30252 (2013.01)] 12 Claims
OG exemplary drawing
 
1. A method comprising:
simulating a simulated agent's traversal of a simulated path in a real environment, including simulating an autonomous vehicle in a simulated traversal of a roadway;
prior to simulation of the simulated agent's traversal of the simulated path, obtaining a plurality of observations acquired by real entities as they traversed real paths, each observation including an image acquired when a corresponding real entity was in a location corresponding to said image on one of the real paths, wherein at least some of the real paths traverse the roadway, and wherein the real entities comprise a human-operated real vehicle that traversed one of the real paths;
wherein the simulating of the simulated agent's traversal comprises simulating a simulated agent's view to form simulated observations, including simulating a first image for a first location on the simulated path, the simulating of the first image including selecting a second image acquired by one of the real entities at a second location different than the first location, and transforming the second image to form the first image according to a relationship of the first location and the second location, wherein the second image comprises a two-dimensional image and the first image comprises a two-dimensional image, wherein transforming the second image to form the first image comprises forming a second three-dimensional image from the second two-dimensional image, forming a first three-dimensional image from the second three-dimensional image according to the relationship of the first location and the second location, and forming the first two-dimensional image from the first three-dimensional image;
wherein the simulating of the simulated agent's traversal further comprises processing the simulated observations according to a control policy to yield control actions, and causing the simulated agent to transverse the simulated path according to the control actions;
wherein the method further comprises updating the control policy according to the simulated agent's traversal of the simulated path, including updating the policy according to the simulated agent's traversal of a plurality of simulated paths according to simulations of the simulated agent's view in a plurality of environmental conditions.