US 12,437,375 B2
Improving digital image inpainting utilizing plane panoptic segmentation and plane grouping
Yuqian Zhou, Urbana, IL (US); Connelly Barnes, Seattle, WA (US); Sohrab Amirghodsi, Seattle, WA (US); and Elya Shechtman, Seattle, WA (US)
Assigned to Adobe Inc., San Jose, CA (US)
Filed by Adobe Inc., San Jose, CA (US)
Filed on Jun. 14, 2024, as Appl. No. 18/743,497.
Application 18/743,497 is a continuation of application No. 17/520,249, filed on Nov. 5, 2021, granted, now 12,056,857.
Prior Publication US 2024/0331114 A1, Oct. 3, 2024
This patent is subject to a terminal disclaimer.
Int. Cl. G06K 9/00 (2022.01); G06F 18/22 (2023.01); G06N 3/02 (2006.01); G06T 5/77 (2024.01); G06T 7/11 (2017.01); G06T 11/00 (2006.01)
CPC G06T 5/77 (2024.01) [G06F 18/22 (2023.01); G06N 3/02 (2013.01); G06T 7/11 (2017.01); G06T 11/00 (2013.01); G06T 2207/20084 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
generating, using an inpainting model, a preliminary inpainted image by inpainting one or more holes of an initial digital image;
generating, from the preliminary inpainted image, an appearance guide from the preliminary inpainted image by:
detecting surface planes within the preliminary inpainted image utilizing a plane detection model; and
detecting panoptic segments within the preliminary inpainted image utilizing a panoptic segmentation model that distinguishes between instances of objects with a shared semantic label; and
generating a final inpainted image by utilizing a guided inpainting model informed by the appearance guide as a guide for inpainting the one or more holes of the initial digital image.