US 12,080,009 B2
Multi-channel high-quality depth estimation system to provide augmented and virtual realty features
Fangwen Tu, Singapore (SG); and Bo Li, Singapore (SG)
Assigned to Black Sesame Technologies Inc., San Jose, CA (US)
Filed by Black Sesame International Holding Limited, San Jose, CA (US)
Filed on Aug. 31, 2021, as Appl. No. 17/463,188.
Prior Publication US 2023/0063150 A1, Mar. 2, 2023
Int. Cl. G06T 7/11 (2017.01); G06T 5/70 (2024.01); G06T 5/77 (2024.01); G06T 7/13 (2017.01); G06T 7/536 (2017.01)
CPC G06T 7/536 (2017.01) [G06T 5/70 (2024.01); G06T 5/77 (2024.01); G06T 7/11 (2017.01); G06T 7/13 (2017.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/20192 (2013.01)] 16 Claims
OG exemplary drawing
 
1. A multi-channel convolution Neural network (CNN) based depth estimation system for a monocular camera, wherein the depth estimation system comprising:
a monocular depth estimation module, wherein the monocular depth estimation module comprising:
a predictor unit for predicting depth in a single image based on pre-stored images and one or more parameters of the monocular camera; and
an edge alignment quality unit for removing edge discontinuities in the depth of the image by introducing a semantic head during a training to consider semantic objects to generate an aligned image;
a depth map refinement module, wherein the depth map refinement module comprising:
a panoptic segmentation unit for applying one or more semantic labels to one or more portions of the aligned image to generate a segmented image; and
a dictionary unit for applying a depth pattern to each of the one or more portions of the segmented image based on the one or more semantic labels to generate a processed image;
a depth layout module, wherein the depth layout module facilitates the depth map refinement module by providing a depth pattern to one or more unrecognized semantic labels in the processed image to form a labelled image;
a depth inpainting module, wherein the depth inpainting module imprints one or more occluded regions in the labelled image to generate an inpainted image; and
an output module, wherein the output module adds a plurality of augmented reality and virtual reality features to the inpainted image to produce a three-dimensional image.