US 12,033,281 B2
Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking
Suren Deepak Rajasekaran, Milpitas, CA (US); Hiroyuki Takeda, San Jose, CA (US); and Kenji Tashiro, San Jose, CA (US)
Assigned to SONY GROUP CORPORATION, Tokyo (JP); and Sony Corporation of America, New York, NY (US)
Filed by SONY GROUP CORPORATION, Tokyo (JP); and Sony Corporation of America, New York, NY (US)
Filed on Mar. 29, 2022, as Appl. No. 17/706,996.
Claims priority of provisional application 63/169,323, filed on Apr. 1, 2021.
Prior Publication US 2022/0319114 A1, Oct. 6, 2022
Int. Cl. G06T 17/20 (2006.01); G06T 7/20 (2017.01); G06T 7/30 (2017.01); G06V 40/10 (2022.01)
CPC G06T 17/20 (2013.01) [G06T 7/20 (2013.01); G06T 7/30 (2017.01); G06V 40/10 (2022.01); G06T 2200/04 (2013.01)] 21 Claims
OG exemplary drawing
 
1. A method programmed in a non-transitory of a device comprising:
using a volumetric capture system configured for 3D scanning and 4D scanning including capturing photos and video simultaneously, wherein the 3D scanning and 4D scanning includes detecting muscle deformation of an actor; and
implementing mesh generation based on the 3D scanning and 4D scanning,
wherein the 3D scanning and 4D scanning include:
3D scans to be used to generate automatic high-fidelity extreme poses and
4D scans which include high temporal resolution which enables mesh tracking to automatically register extreme pose meshes for blending and mesh generation of a muscle deforming over time.