US 12,008,817 B2
Systems and methods for depth estimation in a vehicle
Albert Shalumov, Petah Tikva (IL); and Michael Slutsky, Kfar Saba (IL)
Assigned to GM GLOBAL TECHNOLOGY OPERATIONS LLC, Detroit, MI (US)
Filed by GM GLOBAL TECHNOLOGY OPERATIONS LLC, Detroit, MI (US)
Filed on Mar. 11, 2021, as Appl. No. 17/198,954.
Prior Publication US 2022/0292289 A1, Sep. 15, 2022
Int. Cl. G06V 20/58 (2022.01); G06F 18/214 (2023.01); G06F 18/22 (2023.01); G06N 3/04 (2023.01); G06N 3/08 (2023.01); G06T 5/70 (2024.01); G06T 7/13 (2017.01); G06T 7/55 (2017.01); G06V 30/262 (2022.01); H04N 7/18 (2006.01); H04N 23/698 (2023.01); H04N 23/90 (2023.01); G05D 1/00 (2006.01)
CPC G06V 20/58 (2022.01) [G06F 18/214 (2023.01); G06F 18/22 (2023.01); G06N 3/04 (2013.01); G06N 3/08 (2013.01); G06T 5/70 (2024.01); G06T 7/13 (2017.01); G06T 7/55 (2017.01); G06V 30/274 (2022.01); H04N 7/181 (2013.01); H04N 23/698 (2023.01); H04N 23/90 (2023.01); G05D 1/0246 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30261 (2013.01)] 17 Claims
OG exemplary drawing
 
1. A method of controlling a vehicle, the method comprising:
receiving, via at least one processor, respective image data from at least two cameras, wherein fields of view of adjacent cameras of the at least two cameras partially overlap and wherein the at least two cameras are mounted to the vehicle;
processing the respective image data through a neural network, wherein the neural network is trained to provide depth data and semantic segmentation data as outputs and wherein the neural network is trained using a loss function that combines a plurality of loss terms including at least a semantic segmentation loss term and a panoramic loss term, wherein the panoramic loss term includes a similarity measure regarding overlapping patches of the respective image data that each correspond to a region of overlapping fields of view of the adjacent cameras and wherein the panoramic loss term is calculated by steps including: projecting the respective image data into a combined equirectangular panoramic image; applying a mask to the combined equirectangular panoramic image so as to extract the overlapping image patches; and performing the similarity measure on the overlapping image patches to quantify the similarity of the overlapping image patches, wherein the plurality of loss terms further includes a disparity smoothness loss term, wherein disparity is inverse of depth;
controlling, via the at least one processor, a function of the vehicle based on the depth data.