US 12,260,572 B2
Single image 3D photography with soft-layering and depth-aware inpainting
Varun Jampani, Rockland, MA (US); Huiwen Chang, Mountain View, CA (US); Kyle Sargent, Cambridge, MA (US); Abhishek Kar, Sunnyvale, CA (US); Richard Tucker, New York City, NY (US); Dominik Kaeser, New York City, NY (US); Brian L. Curless, Seattle, WA (US); David Salesin, Sausalito, CA (US); William T. Freeman, Acton, MA (US); Michael Krainin, Arlington, MA (US); and Ce Liu, Belmont, MA (US)
Assigned to Google LLC, Mountain View, CA (US)
Appl. No. 17/907,529
Filed by Google LLC, Mountain View, CA (US)
PCT Filed Aug. 5, 2021, PCT No. PCT/US2021/044828
§ 371(c)(1), (2) Date Sep. 27, 2022,
PCT Pub. No. WO2023/014368, PCT Pub. Date Feb. 9, 2023.
Prior Publication US 2024/0249422 A1, Jul. 25, 2024
Int. Cl. G06T 7/00 (2017.01); G06T 5/60 (2024.01); G06T 5/77 (2024.01); G06T 7/50 (2017.01)
CPC G06T 7/50 (2017.01) [G06T 5/60 (2024.01); G06T 5/77 (2024.01); G06T 2207/20081 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
obtaining a monocular image having an initial viewpoint;
determining, based on the monocular image, a depth image comprising a plurality of pixels, wherein each respective pixel of the depth image has a corresponding depth value;
determining, for each respective pixel of the depth image, a corresponding depth gradient associated with the respective pixel of the depth image;
determining a foreground visibility map comprising, for each respective pixel of the depth image, a visibility value that is inversely proportional to the corresponding depth gradient;
determining, based on the depth image, a background disocclusion mask comprising, for each respective pixel of the depth image, a disocclusion value indicating a likelihood that a corresponding pixel of the monocular image will be disoccluded by a change in the initial viewpoint;
generating (i) an inpainted image by inpainting, using an inpainting model, portions of the monocular image in accordance with the background disocclusion mask and (ii) an inpainted depth image by inpainting, using the inpainting model, portions of the depth image in accordance with the background disocclusion mask;
generating (i), based on the depth image, a first three-dimensional (3D) representation of the monocular image and (ii), based on the inpainted depth image, a second 3D representation of the inpainted image; and
generating a modified image having an adjusted viewpoint that is different from the initial viewpoint by combining the first 3D representation with the second 3D representation in accordance with the foreground visibility map.