US 12,094,144 B1
Real-time confidence-based image hole-filling for depth maps
Vikram Shree, Cambridge, MA (US); Piotr Swierczynski, Providence, RI (US); and Leaf Alden Jiang, Concord, MA (US)
Assigned to NODAR Inc., Somerville, MA (US)
Filed by NODAR Inc., Somerville, MA (US)
Filed on Jun. 23, 2023, as Appl. No. 18/340,806.
Int. Cl. G06T 7/593 (2017.01); G06T 5/20 (2006.01); G06T 5/77 (2024.01)
CPC G06T 7/593 (2017.01) [G06T 5/20 (2013.01); G06T 5/77 (2024.01); G06T 2207/10012 (2013.01); G06T 2207/10028 (2013.01); G06T 2207/30252 (2013.01)] 21 Claims
OG exemplary drawing
 
1. A method of repairing a 3D image generated by a vision system on a vehicle, the method comprising:
(a) obtaining a three-dimensional (3D) image of a scene, the 3D image corresponding to a depth map of the scene generated from at least one two-dimensional (2D) image captured of the scene, wherein:
the 3D image comprises a plurality of pixels arranged in a matrix H, and
the pixels comprise first pixels and second pixels, each of the first pixels having a first pixel value corresponding to a depth value for a portion of the scene represented by the first pixel, and each of the second pixels corresponding to a hole having an unknown depth value for a portion of the scene represented by the second pixel;
(b) determining a non-binary confidence mask matrix C for the 3D image having a distribution of confidence values;
(c) for each of at least some of the second pixels:
determining a kernel G of neighboring pixels of the second pixel, the neighboring pixels of the kernel G having values corresponding to a predetermined distribution, and
convolving the kernel G with a portion of the confidence mask matrix C and a portion of the matrix H corresponding to the kernel G to obtain an estimated depth value for the second pixel;
(d) generating a corrected matrix Ĥ representing a corrected 3D image, the corrected matrix Ĥ comprising the first pixels and corrected second pixels, wherein the corrected second pixels correspond to the second pixels corrected to have second pixel values corresponding to the estimated depth values, wherein elements of the corrected matrix Ĥ are normalized based on a normalization matrix Ĉ obtained from the confidence mask matrix C; and
outputting data corresponding to at least a portion of the corrected matrix Ĥ to the vehicle,
wherein a total number of holes in the matrix H is greater than a total number of holes in the corrected matrix Ĥ.