US 12,437,548 B1
Vision based object representation
Jonathan Tyler Dowdall, San Francisco, CA (US); Kratarth Goel, Albany, CA (US); Adam Edward Pollack, San Francisco, CA (US); and Scott M. Purdy, Lake Forest Park, WA (US)
Assigned to Zoox, Inc., Foster City, CA (US)
Filed by Zoox, Inc., Foster City, CA (US)
Filed on Dec. 17, 2021, as Appl. No. 17/555,257.
Int. Cl. G06V 20/56 (2022.01); B60W 60/00 (2020.01); G01S 7/48 (2006.01); G01S 17/87 (2020.01); G06T 7/11 (2017.01); G06T 7/50 (2017.01); G06V 10/22 (2022.01); G06V 10/764 (2022.01); G06V 10/774 (2022.01); G06V 10/80 (2022.01)
CPC G06V 20/56 (2022.01) [B60W 60/0015 (2020.02); G01S 7/4802 (2013.01); G01S 17/87 (2013.01); G06T 7/11 (2017.01); G06T 7/50 (2017.01); G06V 10/225 (2022.01); G06V 10/764 (2022.01); G06V 10/774 (2022.01); G06V 10/803 (2022.01); B60W 2420/403 (2013.01); B60W 2420/408 (2024.01); B60W 2554/4029 (2020.02); G06T 2207/20081 (2013.01); G06T 2207/20132 (2013.01); G06T 2207/30252 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system comprising:
one or more processors; and
one or more non-transitory computer readable media storing computer executable instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving two-dimensional image data from an image sensor of an autonomous vehicle;
receiving lidar data from a lidar sensor of the autonomous vehicle, the lidar data associated with a portion of the two-dimensional image data;
determining, based at least in part on the two-dimensional image data and independent of requiring the lidar data, a three-dimensional (3D) bounding box associated with an object;
combining, as combined depth data, first depth information associated with the two-dimensional image data with second depth information associated with the lidar data;
determining, based at least in part on the lidar data and applying a depth completion algorithm to the combined depth data, dense depth data including third depth information having a higher density than the second depth information associated with the lidar data;
inputting the 3D bounding box and the dense depth data into a trained machine learned model;
receiving, from the trained machine learned model, updated 3D bounding box data associated with the object, the updated 3D bounding box data determined based at least in part on the 3D bounding box and the dense depth data; and
controlling the autonomous vehicle based at least in part on the updated 3D bounding box data associated with the object.