US 12,450,768 B2
End-to-end tracking of objects
Davi Eugenio Nascimento Frossard, Toronto (CA); and Raquel Urtasun, Toronto (CA)
Assigned to Aurora Operations, Inc., Pittsburgh, PA (US)
Filed by Aurora Operations, Inc., Pittsburgh, PA (US)
Filed on May 24, 2021, as Appl. No. 17/328,566.
Application 17/328,566 is a continuation of application No. 16/122,203, filed on Sep. 5, 2018, granted, now 11,017,550, issued on May 25, 2021.
Claims priority of provisional application 62/586,700, filed on Nov. 15, 2017.
Prior Publication US 2021/0362596 A1, Nov. 25, 2021
Int. Cl. B60K 31/00 (2006.01); G01S 17/89 (2020.01); G06N 3/045 (2023.01); G06N 3/08 (2023.01); G06T 7/20 (2017.01); G06T 7/246 (2017.01); G06T 7/70 (2017.01); G06T 7/90 (2017.01); G06V 20/58 (2022.01)
CPC G06T 7/70 (2017.01) [B60K 31/0008 (2013.01); G01S 17/89 (2013.01); G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06T 7/20 (2013.01); G06T 7/248 (2017.01); G06T 7/90 (2017.01); G06V 20/58 (2022.01); B60K 2031/0016 (2013.01); G06T 2207/10024 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30241 (2013.01); G06T 2207/30252 (2013.01)] 18 Claims
OG exemplary drawing
 
1. An autonomous vehicle computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the one or more processors to perform operations, the operations comprising:
receiving, through one or more sensors of an autonomous vehicle, sensor data associated with an environment, the sensor data comprising RGB data and one or more LIDAR point clouds;
determining an object detection associated with an object within the environment based at least in part on one or more first models and the sensor data;
determining a detection score for the object detection, wherein determining a detection score comprises:
fusing the one or more LIDAR point clouds with the RGB data by extracting one or more features from the RGB data; and
encoding one or more binary parameters associated with the object detection, wherein at least one binary parameter of the one or more binary parameters is indicative of whether the object detection is associated with a trajectory beginning or a trajectory ending;
the one or more first models comprising one or more first machine-learned models configured to detect the object in the environment based at least in part on the sensor data being input into the one or more first models and configured to determine the detection score for the object detection based on the one or more binary parameters;
tracking the object detection over a sequence of sensor data inputs based at least in part on one or more second models,
the one or more second models comprising one or more second machine-learned models configured to track the object over the sequence of sensor data inputs, the one or more second models being different from the one or more first models; and
generating a trajectory for the object based at least in part on one or more linear constraints configured to link the object detection over the sequence of sensor data inputs.