	US 11,995,761 B2
	Methods and system for generating virtual sensor data of a virtual single-photon avalanche diode (SPAD) lidar sensor of a virtual vehicle simulator
Prasanna Sivakumar, Pittsburgh, PA (US); Kris Kitani, Pittsburgh, PA (US); Matthew O'Toole, Pittsburgh, PA (US); Xinshuo Weng, Pittsburgh, PA (US); Shawn Hunt, Bethel Park, PA (US); and Yunze Man, Pittsburgh, PA (US)
Assigned to DENSO CORPORATION, Kariya (JP); and Carnegie Mellon University, Pittsburgh, PA (US)
Filed by DENSO CORPORATION, Kariya (JP); and Carnegie Mellon University, Pittsburgh, PA (US)
Filed on Mar. 30, 2022, as Appl. No. 17/708,760.
Claims priority of provisional application 63/262,213, filed on Oct. 7, 2021.
Prior Publication US 2023/0114731 A1, Apr. 13, 2023
Int. Cl. G06T 15/20 (2011.01)

CPC G06T 15/205 (2013.01) [G06T 2210/56 (2013.01)]

15 Claims

1. A method of generating virtual sensor data of a virtual single-photon avalanche diode (SPAD) lidar sensor, comprising:

generating a two-dimensional (2D) lidar array having a plurality of cells, wherein each cell is associated with a coordinate indicative of a position of the cell in the 2D lidar array;

interpolating image data from a virtual camera with the 2D lidar array to define auxiliary image data, wherein the auxiliary image data correlates the plurality of cells of the 2D lidar array with a plurality of pixels of the image data;

generating a virtual ambient image based on a red-channel (R-channel) data of the auxiliary image data;

identifying a plurality of virtual echoes of the virtual SPAD lidar sensor based on the R-channel data and a defined photon threshold;

defining a virtual point cloud indicative of virtual photon measurements of the virtual SPAD lidar sensor based on the plurality of virtual echoes;

determining a virtual photon measurement for each cell of the plurality of cells based on the R-channel data of the auxiliary image data; and

determining a virtual fill factor of the virtual SPAD lidar sensor based on the virtual photon measurements and a Gaussian kernel-based model, wherein the plurality of virtual echoes is identified based on the virtual fill factor of the virtual SPAD lidar sensor and the defined photon threshold;

determining the virtual photon measurement further comprises, for each cell of the plurality of cells:

determining a number of signal photons for a selected time interval based on a signal photon model and the R-channel data;

determining a number of ambient photons based on a normalization of the R-channel over the auxiliary image data; and

generate a three-dimensional (3D) tensor photon count for the selected time interval based on the number of signal photons and the number of ambient photons;

outputting data indicative of the virtual ambient image, the virtual photon measurements, the virtual point cloud, or a combination thereof, as a virtual sensor data of the virtual SPAD lidar sensor.