US 12,367,640 B2
Virtual role-based multimodal interaction method, apparatus and system, storage medium, and terminal
Jinxiang Chai, Shanghai (CN); Hongbing Tan, Shanghai (CN); Xingtang Xiong, Shanghai (CN); Congyi Wang, Shanghai (CN); Zhiqiang Liang, Shanghai (CN); Bin Wang, Shanghai (CN); and Yu Chen, Shanghai (CN)
Assigned to MOFA (SHANGHAI) INFORMATION TECHNOLOGY CO., LTD., Shanghai (CN); and SHANGHAI MOVU TECHNOLOGY CO., LTD., Shanghai (CN)
Appl. No. 18/024,035
Filed by MOFA (SHANGHAI) INFORMATION TECHNOLOGY CO., LTD., Shanghai (CN); and SHANGHAI MOVU TECHNOLOGY CO., LTD., Shanghai (CN)
PCT Filed Aug. 9, 2021, PCT No. PCT/CN2021/111422
§ 371(c)(1), (2) Date Feb. 28, 2023,
PCT Pub. No. WO2022/048403, PCT Pub. Date Mar. 10, 2022.
Claims priority of application No. 202010906582.5 (CN), filed on Sep. 1, 2020.
Prior Publication US 2023/0316643 A1, Oct. 5, 2023
Int. Cl. G06T 17/00 (2006.01); G06T 13/40 (2011.01); G06V 40/16 (2022.01); G06V 40/18 (2022.01); G06V 40/20 (2022.01)
CPC G06T 17/00 (2013.01) [G06T 13/40 (2013.01); G06V 40/171 (2022.01); G06V 40/174 (2022.01); G06V 40/193 (2022.01); G06V 40/20 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A virtual role-based multimodal interaction method, comprising:
acquiring input information, wherein the input information comprises one or more data types;
inputting the input information into a perception layer to enable the perception layer to recognize and process the input information according to a data type of the input information to obtain a recognition result;
inputting the recognition result into a logic decision-making layer to enable the logic decision-making layer to process the recognition result and generate a drive instruction corresponding to the recognition result;
acquiring multimodal virtual content according to the drive instruction, wherein the multimodal virtual content at least comprises a virtual role; and
outputting the acquired multimodal virtual content;
wherein the logic decision-making layer comprises a logic decision-making model, the logic decision-making model is trained based on training samples, the training samples are samples of the recognition result and samples of the drive instruction, and the training samples comprise a training set and a test set.