US 12,190,484 B2
Generating modified digital images using deep visual guided patch match models for image inpainting
Sohrab Amirghodsi, Seattle, WA (US); Lingzhi Zhang, Philadelphia, PA (US); Zhe Lin, Fremont, CA (US); Connelly Barnes, Seattle, WA (US); and Elya Shechtman, Seattle, WA (US)
Assigned to Adobe Inc., San Jose, CA (US)
Filed by Adobe Inc., San Jose, CA (US)
Filed on Mar. 15, 2021, as Appl. No. 17/202,019.
Prior Publication US 2022/0292650 A1, Sep. 15, 2022
Int. Cl. G06T 5/77 (2024.01); G06N 3/08 (2023.01); G06T 3/4053 (2024.01); G06T 7/11 (2017.01); G06T 7/50 (2017.01)
CPC G06T 5/77 (2024.01) [G06N 3/08 (2013.01); G06T 3/4053 (2013.01); G06T 7/11 (2017.01); G06T 7/50 (2017.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to replace pixels within a region of a digital image to generate a modified digital image by:
generating an inpainted digital image from the digital image utilizing an inpainting neural network, the inpainted digital image comprising an initial set of replacement pixels used to fill a region of the digital image;
generating, from the inpainted digital image utilizing a visual guide algorithm, a deep visual guide that informs patch match inpainting;
identifying replacement pixels for inpainting the region of the digital image by utilizing a patch match model with a weighted cost function to identify the replacement pixels according to a weighted combination of a structure image guide, an image depth guide, and a segmentation image guide indicated by the deep visual guide for potential replacement pixels, wherein the weighted cost function represents relationships between the pixels within the region of the digital image and the potential replacement pixels within additional regions of the digital image based on a modification of an initial cost function of the patch match model; and
generating, utilizing the patch match model, the modified digital image by replacing the region of the digital image with the replacement pixels.