US 12,243,251 B1
Robust consistent video depth estimation
Johannes Peter Kopf, Medina, WA (US); Xuejian Rong, Seattle, WA (US); and Jia-Bin Huang, Blacksburg, VA (US)
Assigned to Meta Platforms, Inc., Menlo Park, CA (US)
Filed by Meta Platforms, Inc., Menlo Park, CA (US)
Filed on Dec. 10, 2021, as Appl. No. 17/548,307.
Int. Cl. G06T 7/55 (2017.01); G06T 7/11 (2017.01); G06T 7/30 (2017.01); G06T 7/70 (2017.01)
CPC G06T 7/55 (2017.01) [G06T 7/11 (2017.01); G06T 7/30 (2017.01); G06T 7/70 (2017.01); G06T 2207/10016 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/30244 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
receiving a video comprising a plurality of image frames;
generating, based on a first image frame of the plurality of image frames, an initial depth map using a machine-learning model;
determining a misalignment error indicating depth misalignments in the initial depth map using a reprojection technique;
generating one or more optimized camera parameters and a flexible deformation spline associated with the first image frame to minimize the misalignment error; and
generating a refined depth map by adjusting the initial depth map associated with the first image frame using the flexible deformation spline associated with the first image frame, wherein the flexible deformation spline comprises a grid in which one or more intersections of the grid are deformed outward or inward along a ray cast from the one or more optimized camera parameters to the grid.