US 12,254,784 B2
Emotional evolution method and terminal for virtual avatar in educational metaverse
Dejian Liu, Fuzhou (CN); Yan Ge, Fuzhou (CN); Jing Peng, Wuhan (CN); Zheng Zhong, Wuhan (CN); and Jian Xu, Wuhan (CN)
Assigned to Fujian TQ Digital Inc, Fuzhou (CN); and Central China Normal University, Wuhan (CN)
Filed by Fujian TQ Digital Inc., Fuzhou (CN); and Central China Normal University, Wuhan (CN)
Filed on Sep. 27, 2023, as Appl. No. 18/475,569.
Claims priority of application No. 202310815919.5 (CN), filed on Jul. 5, 2023.
Prior Publication US 2025/0014470 A1, Jan. 9, 2025
Int. Cl. G09B 5/06 (2006.01); G06T 13/40 (2011.01); G06V 40/16 (2022.01)
CPC G09B 5/065 (2013.01) [G06T 13/40 (2013.01); G06V 40/176 (2022.01)] 8 Claims
OG exemplary drawing
 
1. An emotional evolution method for a virtual avatar in educational metaverse, comprising:
collecting expression data and audio data of the virtual avatar, and performing emotional feature extraction based on the expression data and the audio data to obtain a sound emotional feature and an expression emotional feature, wherein the expression data of the virtual avatar includes data associated with a facial expression of the virtual avatar, the facial expression of the virtual avatar including at least one of smiling, mouth opening, staring, mouth contraction, calming, and squinting;
fusing the sound emotional feature with the expression emotional feature by using an emotional feature fusion model to obtain a multi-modal emotional feature fusion result, and performing emotion recognition on the multi-modal emotional feature fusion result to obtain an emotional category corresponding to the multi-modal emotional feature fusion result;
determining a semantic vector of the emotional category, and labeling the multi-modal emotional feature fusion result based on the semantic vector of the emotional category to generate an emotional evolution sequence for the virtual avatar, wherein the emotional evolution sequence is an algorithm that enables the virtual avatar to change its facial expression in an intelligent teaching system; and
extracting a target emotional evolution pattern from the emotional evolution sequence, and driving the virtual avatar to perform emotional expression according to the target emotional evolution pattern;
wherein fusing the sound emotional feature with the expression emotional feature by using the emotional feature fusion model to obtain the multi-modal emotional feature fusion result comprises:
respectively normalizing the sound emotional feature and the expression emotional feature to obtain a sound emotional feature vector and an expression emotional feature vector;
calculating the similarity between the sound emotional feature vector and the expression emotional feature vector by using a Chebyshev distance;
calculating a weight ratio of each vector by using an attention mechanism according to the similarity and based on the sound emotional feature vector and the expression emotional feature vector, wherein each vector is the sound emotional feature vector or the expression emotional feature vector;
obtaining sound emotional feature vector representation and expression emotional feature vector representation according to the weight ratio of each vector, the sound emotional feature vector and the expression emotional feature vector; and
inputting the sound emotional feature vector representation and the expression emotional feature vector representation to the emotional feature fusion model, and outputting the multi-modal emotional feature fusion result,
wherein labeling the multi-modal emotional feature fusion result based on the semantic vector of the emotional category to generate an emotional evolution sequence comprises:
labeling the multi-modal emotional feature fusion result by using the semantic vector of the emotional category to generate an emotional semantic sequence;
calculating fitness among different emotional semantics in the emotional semantic sequence by using a kernel function; and
determining whether the fitness is lower than a preset fitness threshold, if yes, amending the emotional semantics corresponding to the fitness by using a semantic rewriting algorithm to obtain an amended emotional semantic sequence, and generating the emotional evolution sequence based on the amended emotional semantic sequence by using a time sequence analysis algorithm, and if not, generating the emotional evolution sequence based on the emotional semantic sequence by using the time sequence analysis algorithm.