US 11,935,250 B2
Method, device and computer-readable storage medium with instructions for processing sensor data
Simon Steinmeyer, Braunschweig (DE); Marek Musial, Berlin (DE); Carsten Deeg, Berlin (DE); Thorsten Bagdonat, Braunschweig (DE); and Thorsten Graf, Gifhorn (DE)
Assigned to VOLKSWAGEN AKTIENGESELLSCHAFT, Wolfsburg (DE)
Appl. No. 17/048,701
Filed by Volkswagen Aktiengesellschaft, Wolfsburg (DE)
PCT Filed Mar. 27, 2019, PCT No. PCT/EP2019/057701
§ 371(c)(1), (2) Date Oct. 19, 2020,
PCT Pub. No. WO2019/201565, PCT Pub. Date Oct. 24, 2019.
Claims priority of application No. 10 2018 205 879.2 (DE), filed on Apr. 18, 2018.
Prior Publication US 2021/0158544 A1, May 27, 2021
Int. Cl. G06T 7/269 (2017.01); G01S 13/86 (2006.01); G01S 13/931 (2020.01)
CPC G06T 7/269 (2017.01) [G01S 13/867 (2013.01); G01S 13/931 (2013.01); G06T 2207/10028 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30252 (2013.01)] 17 Claims
OG exemplary drawing
 
1. Method for processing sensor data, comprising:
obtaining camera images by a camera including a first camera image captured at a first time and a second image captured at a second time;
detecting 3D-checkpoints by at least one 3D-sensor at a 3D-checkpoint detection time after the first time of the first camera image and before the second time of the second camera image; and
fusing the camera images with the 3D-checkpoints to form data of a virtual sensor, wherein fusing the camera images with the 3D-checkpoints comprises:
calculating an optical flow from at least the first camera image and the second camera image;
selecting either the first camera image or the second camera image based on whether the 3D-checkpoint detection time is closer in time to the first time of the first camera image or the second time of the second camera image;
converting the selected first or second camera image to the 3D-checkpoint detection time based at least on the optical flow and a time difference between the respective first time or second time of the selected first or second camera image and the 3D-checkpoint detection time; and
projecting the 3D-checkpoints into the converted selected first or second selected camera image.