US 12,076,148 B2
Systems and methods for automated recognition of bodily expression of emotion
James Z. Wang, State College, PA (US); Yu Luo, State College, PA (US); Jianbo Ye, State College, PA (US); and Reginald B. Adams, Pennsylvania Furnace, PA (US)
Assigned to THE PENN STATE RESEARCH FOUNDATION, University Park (PA)
Filed by THE PENN STATE RESEARCH FOUNDATION, University Park, PA (US)
Filed on Jul. 1, 2020, as Appl. No. 16/918,604.
Claims priority of provisional application 62/870,901, filed on Jul. 5, 2019.
Prior Publication US 2021/0000404 A1, Jan. 7, 2021
Int. Cl. A61B 5/16 (2006.01); A61B 5/00 (2006.01); A61B 5/11 (2006.01); A61B 5/117 (2016.01); G16H 10/60 (2018.01); G16H 20/70 (2018.01); G16H 50/50 (2018.01)
CPC A61B 5/165 (2013.01) [A61B 5/1128 (2013.01); A61B 5/117 (2013.01); A61B 5/7264 (2013.01); A61B 5/7275 (2013.01); G16H 10/60 (2018.01); G16H 20/70 (2018.01); G16H 50/50 (2018.01)] 20 Claims
OG exemplary drawing
 
1. An emotion analysis and recognition system, comprising:
a query system configured to interact with a user; and
an automated recognition of bodily expression of emotion (ARBEE) system communicatively coupled to the query system, the ARBEE system having a processor and a memory storing program instructions, the program instructions executable by the processor to:
receive a plurality of body movement models, each body movement model generated based on a crowdsourced body language dataset;
calculate at least one evaluation metric for each body movement model, the at least one evaluation metric to evaluate a classification performance for each of a plurality of categorical emotions and/or to evaluate regression performance for each of a plurality of dimensional emotions, the categorical and dimensional emotions being determined based on a combination of movements for body parts of subjects, facial expressions of the subjects, associated audio, associated context, and a pixel analysis of cropped images of a human body;
select a highest ranked body movement model based on the at least one evaluation metric calculated for each body movement model;
combine the highest ranked body movement model with at least one other body movement model of the plurality of body movement models;
calculate at least one evaluation metric for each combination of body movement models;
determine a highest ranked combination of body movement models;
based on the determined highest ranked combination of body movement models, predict a bodily expression of emotion;
receive, from the query system, a video;
apply the received video as input to the highest ranked combination of body movement models; and
transmit an emotion result to the query system, the emotion result providing information usable by the query system for determining an emotion of a target,
wherein the query system interacts with the user based on the emotion result received from the ARBEE system.