US 11,657,522 B2
Sparse auxiliary network for depth completion
Vitor Guizilini, Santa Clara, CA (US); Rares A. Ambrus, San Francisco, CA (US); and Adrien David Gaidon, Mountain View, CA (US)
Assigned to Toyota Research Institute, Inc., Los Altos, CA (US)
Filed by Toyota Research Institute, Inc., Los Altos, CA (US)
Filed on Jan. 7, 2021, as Appl. No. 17/143,684.
Claims priority of provisional application 63/112,234, filed on Nov. 11, 2020.
Prior Publication US 2022/0148202 A1, May 12, 2022
Int. Cl. G06V 10/40 (2022.01); G06V 20/56 (2022.01); G01B 11/22 (2006.01); G01S 17/08 (2006.01); G01S 17/89 (2020.01); G06T 7/50 (2017.01); B60W 60/00 (2020.01); G01C 21/00 (2006.01); B60W 30/095 (2012.01); G06T 9/00 (2006.01)
CPC G06T 7/50 (2017.01) [B60W 30/0956 (2013.01); B60W 60/001 (2020.02); G01B 11/22 (2013.01); G01C 21/3815 (2020.08); G01S 17/08 (2013.01); G01S 17/89 (2013.01); G06T 9/00 (2013.01); G06V 10/40 (2022.01); G06V 20/56 (2022.01); B60W 2420/42 (2013.01); B60W 2420/52 (2013.01); B60W 2554/20 (2020.02); B60W 2554/4029 (2020.02); G06T 2207/10028 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30252 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A depth system, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing:
a network module including instructions that, when executed by the one or more processors, cause the one or more processors to:
generate depth features from depth data using a sparse auxiliary network (SAN) by i) sparsifying the depth data, ii) applying sparse residual blocks of the SAN to the depth data, and iii) densifying the depth features;
generate a depth map from the depth features and a monocular image that corresponds with the depth data according to a depth model that includes the SAN; and
provide the depth map as depth estimates of objects represented in the monocular image.