US 12,248,093 B2
System and method for training a neural network to perform object detection using lidar sensors and radar sensors
Prasanna Sivakumar, Pittsburgh, PA (US); Shawn Hunt, Bethel Park, PA (US); Kris Kitani, Pittsburgh, PA (US); Matthew O'Toole, Pittsburgh, PA (US); and Yu-Jhe Li, Pittsburgh, PA (US)
Assigned to DENSO CORPORATION, Kariya (JP); and Carnegie Mellon University, Pittsburgh, PA (US)
Filed by DENSO CORPORATION, Kariya (JP); and Carnegie Mellon University, Pittsburgh, PA (US)
Filed on Jun. 6, 2022, as Appl. No. 17/832,906.
Claims priority of provisional application 63/263,103, filed on Oct. 27, 2021.
Prior Publication US 2023/0130588 A1, Apr. 27, 2023
Int. Cl. G01S 7/41 (2006.01); G01S 13/89 (2006.01); G01S 17/89 (2020.01); G06N 3/045 (2023.01); G06V 20/58 (2022.01)
CPC G01S 7/417 (2013.01) [G01S 13/89 (2013.01); G01S 17/89 (2013.01); G06N 3/045 (2023.01); G06V 20/58 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A method for training a student neural network to detect one or more objects based on radar data obtained from one or more radar sensors and lidar data obtained from one or more lidar sensors, the method comprising:
generating a radar-based intensity map based on the radar data and a lidar-based intensity map based on the lidar data;
performing one or more augmentation routines on the radar-based intensity map and the lidar-based intensity map to generate a radar input and a lidar input;
generating, using a teacher neural network, a plurality of teacher-based bounding boxes based on the radar input and the lidar input;
generating, using the student neural network, a plurality of student-based bounding boxes based on the radar input and the lidar input;
determining a loss value of the plurality of student-based bounding boxes based on the plurality of teacher-based bounding boxes and a plurality of ground truth bounding boxes;
updating one or more weights of the student neural network based on the loss value; and
updating one or more weights of the teacher neural network based on a moving average associated with the one or more weights of the student neural network.