US 12,452,385 B2
Method and system for deep learning based face swapping with multiple encoders
Jacek Krzysztof Naruniec, Zürich (CH); Romann Matthew Weber, Zürich (CH); and Christopher Richard Schroers, Zürich (CH)
Assigned to Disney Enterprises, Inc., Burbank, CA (US)
Filed by DISNEY ENTERPRISES, INC, Burbank, CA (US)
Filed on Mar. 29, 2022, as Appl. No. 17/707,785.
Prior Publication US 2023/0319223 A1, Oct. 5, 2023
Int. Cl. G06V 40/16 (2022.01); H04N 5/272 (2006.01)
CPC H04N 5/272 (2013.01) [G06V 40/166 (2022.01); G06V 40/176 (2022.01); H04N 2005/2726 (2013.01)] 22 Claims
OG exemplary drawing
 
1. A computer-implemented method of changing a face within an output image or video frame, the method comprising:
receiving a two-dimensional input image that includes a face presenting a facial expression in a pose;
separately encoding different portions of the two-dimensional input image by, for each separately encoded portion, generating a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the two-dimensional input image;
applying an adjustment vector to a multi-dimensional vector of the plurality of multi-dimensional vectors, thereby creating an adjusted vector;
concatenating the plurality of multi-dimensional vectors, including the adjusted vector, into a combined latent space vector; and
decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the two-dimensional input image.