US 12,148,082 B2
System and method for animating an avatar in a virtual world
Yanjie Song, Hong Kong (CN); Leung Ho Philip Yu, Hong Kong (CN); Chi Kin John Lee, Hong Kong (CN); Kaiyi Wu, Hong Kong (CN); and Jiaxin Cao, Hong Kong (CN)
Assigned to The Education University of Hong Kong, Hong Kong (CN)
Filed by The Education University of Hong Kong, Hong Kong (CN)
Filed on Aug. 23, 2022, as Appl. No. 17/893,505.
Prior Publication US 2023/0410398 A1, Dec. 21, 2023
Int. Cl. G06T 13/40 (2011.01); G06T 7/246 (2017.01); G06V 40/16 (2022.01); G06V 40/20 (2022.01)
CPC G06T 13/40 (2013.01) [G06T 7/246 (2017.01); G06V 40/176 (2022.01); G06V 40/23 (2022.01); G06V 40/28 (2022.01)] 12 Claims
OG exemplary drawing
 
7. A method for real-time animating an avatar in a virtual world comprising the steps of:
processing a stream of images capturing an active user to obtain an activity data set arranged to track a pose of the user using individual Artificial Intelligence (AI) models arranged to process the images to identify specific objects of interests to identify specific limbs or body portions from the images and tracking each of these limbs or body portions estimation models, and with the identifications and tracking, to track specific points of these limbs and body parts to construct a skeleton of a moving person so as to in turn determine the pose of the user;
in light of additional input data comprising a preference to match numbers or position of joints on the skeleton and to produce parameters to control movements: recalculating the extracted outputs of the image processor and the estimation models to align with joints of the skeleton and infer missing joints, reformatting the specific points of estimation models to match the skeleton by converting a coordinate of position produced in estimation models from a 4-dimensional to a 3-dimensional matrix, which is available to be a joint of the avatar; making estimates for missing data to fulfil animation requirement; and recognizing and categorizing facial expressions of real users to make an avatar's facial expressions appear more natural;
detecting hand or finger positions of the user so as to identify specific gestures as presented by hands or fingers; and a facial expression detection model arranged to identify facial expression as expressed by the user;
detecting one or more facial expressions of the user; and,
animating the avatar in the virtual world with the plurality of motion tracking points and the detected one or more facial expressions so as to mirror the actions and facial expressions of the active user, wherein the avatar animation engine includes a physics engine arranged to apply an inverse kinematics function onto the plurality of motion tracking points of seven body portions, including the body, left arm, right arm, left leg, right leg, left-hand fingers and right-hand fingers.