US 12,333,817 B2
Association of camera images and radar data in autonomous vehicle applications
Ruichi Yu, Mountain View, CA (US); Shiwei Sheng, Cupertino, CA (US); Kang Li, Sammamish, WA (US); and Xu Chen, Livermore, CA (US)
Assigned to Waymo LLC, Mountain View, CA (US)
Filed by Waymo LLC, Mountain View, CA (US)
Filed on Aug. 3, 2021, as Appl. No. 17/444,338.
Prior Publication US 2023/0038842 A1, Feb. 9, 2023
Int. Cl. G06V 20/56 (2022.01); G01S 7/41 (2006.01); G01S 13/86 (2006.01); G01S 13/931 (2020.01); G06F 18/25 (2023.01); G06N 20/20 (2019.01)
CPC G06V 20/56 (2022.01) [G01S 7/417 (2013.01); G01S 13/867 (2013.01); G01S 13/931 (2013.01); G06F 18/251 (2023.01); G06N 20/20 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
obtaining, by a processing device, radar data associated with a radar frame of a plurality of radar frames temporally spaced by a frame interval, wherein the radar frame is associated with a first timestamp and depicts a first object in an environment of an autonomous vehicle (AV);
obtaining, by the processing device, a camera image, wherein the camera image is associated with a second timestamp and depicts a second object in the environment of the AV, and wherein a difference between the first timestamp and the second timestamp does not exceed the frame interval;
processing, using a first machine-learning model (MLM), at least the radar data to obtain a first embedding vector associated with the first object;
processing, using a second MLM, at least the camera image to obtain a second embedding vector associated with the second object; and
determining, using the first embedding vector and the second embedding vector, a prediction measure representing a likelihood that the first object depicted in the radar frame and the second object in the camera image correspond to a same object in the environment of the AV.