US 12,406,340 B2
Object class inpainting in digital images utilizing class-specific inpainting neural networks
Haitian Zheng, Rochester, NY (US); Zhe Lin, Fremont, CA (US); Jingwan Lu, Santa Clara, CA (US); Scott Cohen, Sunnyvale, CA (US); Elya Shechtman, Seattle, WA (US); Connelly Barnes, Seattle, WA (US); Jianming Zhang, Campbell, CA (US); Ning Xu, Milpitas, CA (US); and Sohrab Amirghodsi, Seattle, WA (US)
Assigned to Adobe Inc., San Jose, CA (US)
Filed by Adobe Inc., San Jose, CA (US)
Filed on May 13, 2022, as Appl. No. 17/663,317.
Prior Publication US 2023/0368339 A1, Nov. 16, 2023
Int. Cl. G06T 5/00 (2024.01); G06N 3/04 (2023.01); G06T 5/77 (2024.01); G06T 7/11 (2017.01)
CPC G06T 5/77 (2024.01) [G06N 3/04 (2013.01); G06T 7/11 (2017.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving, via a user interface of a client device, an indication of a replacement region of a digital image and a target object class;
generating replacement pixels for the replacement region utilizing a class-specific inpainting neural network corresponding to the target object class and having a plurality of cascaded modulation layers, wherein a given cascaded modulation layer comprises a global modulation block and a spatial modulation block, by:
utilizing the global modulation blocks of the plurality of cascaded modulation layers to apply a modulation based on a global feature code to capture global predictions; and
utilizing the spatial modulation blocks of the plurality of cascaded modulation layers to apply a spatial modulation to refine the global predictions; and
providing, for display via the client device, an inpainted digital image comprising the replacement pixels such that the inpainted digital image portrays an instance of the target object class within the replacement region.