US 11,756,210 B2
Video inpainting via machine-learning models with motion constraints
Oliver Wang, Seattle, WA (US); Matthew Fisher, San Francisco, CA (US); John Nelson, Seattle, WA (US); Geoffrey Oxholm, Albany, CA (US); Elya Shechtman, Seattle, WA (US); and Wenqi Xian, New York, NY (US)
Assigned to Adobe Inc., San Jose, CA (US)
Filed by Adobe Inc., San Jose, CA (US)
Filed on Mar. 12, 2020, as Appl. No. 16/817,100.
Prior Publication US 2021/0287007 A1, Sep. 16, 2021
Int. Cl. G06V 20/40 (2022.01); G06T 7/246 (2017.01); G06N 20/00 (2019.01); G06V 10/44 (2022.01); G06V 10/56 (2022.01)
CPC G06T 7/246 (2017.01) [G06N 20/00 (2019.01); G06V 10/44 (2022.01); G06V 10/56 (2022.01); G06V 20/41 (2022.01); G06V 20/46 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A method in which one or more processing devices performs operations comprising:
accessing a scene depicting a reference object, the scene comprising video frames and an annotation identifying a target region to be modified in one or more of the video frames;
accessing a set of sparse feature points defining a three-dimensional model of the reference object, wherein the set of sparse feature points are computed from changes in position of the reference object across the video frames due to motion of the reference object across the video frames;
determining, from the three-dimensional model of the reference object, a motion constraint comprising a reference motion computed from the set of sparse feature points;
computing a target motion of a target pixel of the target region, wherein the target motion is subject to the motion constraint, wherein the target region occludes, at least in part, the reference object, wherein computing the target motion of the target pixel comprises:
identifying boundary pixels of the target region including a first boundary pixel and a second boundary pixel on an opposite side of the target region from the first boundary pixel;
determining, across the video frames, a first boundary motion of the first boundary pixel and a second boundary motion of the second boundary pixel, wherein the target motion of the target pixel is determined based on the first boundary motion of the first boundary pixel and the second boundary motion of the second boundary pixel; and
updating color data of the target pixel based on the target motion.