	US 11,734,888 B2
	Real-time 3D facial animation from binocular video
Chen Cao, Pittsburgh, PA (US); Vasu Agrawal, Pittsburgh, PA (US); Fernando De la Torre, Pittsburgh, PA (US); Lele Chen, Rochester, NY (US); Jason Saragih, Pittsburgh, PA (US); Tomas Simon Kreuz, Pittsburgh, PA (US); and Yaser Sheikh, Pittsburgh, PA (US)
Assigned to Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed by Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed on Aug. 6, 2021, as Appl. No. 17/396,534.
Claims priority of provisional application 63/179,087, filed on Apr. 23, 2021.
Prior Publication US 2022/0358719 A1, Nov. 10, 2022
Int. Cl. G06T 17/20 (2006.01); G06T 7/586 (2017.01); G06T 7/73 (2017.01); G06V 40/16 (2022.01); G06T 13/40 (2011.01); G06T 15/04 (2011.01); G06N 3/045 (2023.01)

CPC G06T 17/20 (2013.01) [G06N 3/045 (2023.01); G06T 7/586 (2017.01); G06T 7/73 (2017.01); G06T 13/40 (2013.01); G06T 15/04 (2013.01); G06V 40/165 (2022.01); G06V 40/176 (2022.01)]

17 Claims

1. A computer-implemented method, comprising:

collecting multiple images of a subject, the images from the subject comprising one or more simultaneous views from different profiles of the subject;

forming a three-dimensional mesh for the subject based on a facial expression code and a head pose of the subject extracted from the images of the subject;

forming a texture transformation based on an illumination parameter associated with an illumination configuration for the images from the subject;

forming a three-dimensional model for the subject based on the three-dimensional mesh and the texture transformation;

determining a loss factor based on selected points in a test image from the subject and a rendition of the test image by the three-dimensional model; and

updating the three-dimensional model according to the loss factor, wherein collecting multiple images of the subject comprises selecting the illumination configuration based on a position, an intensity, and a color gamut for each of multiple light sources around the subject.