US 12,254,692 B2
Construction method and system of descriptive model of classroom teaching behavior events
Sannyuya Liu, Hubei (CN); Zengzhao Chen, Hubei (CN); Zhicheng Dai, Hubei (CN); Shengming Wang, Hubei (CN); Xiuling He, Hubei (CN); and Baolin Yi, Hubei (CN)
Assigned to CENTRAL CHINA NORMAL UNIVERSITY, Hubei (CN)
Appl. No. 18/011,847
Filed by CENTRAL CHINA NORMAL UNIVERSITY, Hubei (CN)
PCT Filed Sep. 7, 2021, PCT No. PCT/CN2021/116820
§ 371(c)(1), (2) Date Dec. 21, 2022,
PCT Pub. No. WO2023/019652, PCT Pub. Date Feb. 23, 2023.
Claims priority of application No. 202110939047.4 (CN), filed on Aug. 16, 2021.
Prior Publication US 2023/0334862 A1, Oct. 19, 2023
Int. Cl. G06V 20/40 (2022.01); G06Q 50/20 (2012.01); G06V 10/774 (2022.01); G06V 40/20 (2022.01); G10L 25/57 (2013.01); G10L 25/78 (2013.01)
CPC G06V 20/44 (2022.01) [G06Q 50/205 (2013.01); G06V 10/774 (2022.01); G06V 20/41 (2022.01); G06V 20/49 (2022.01); G06V 40/20 (2022.01); G10L 25/57 (2013.01); G10L 25/78 (2013.01)] 8 Claims
OG exemplary drawing
 
1. A construction method of a descriptive model of classroom teaching behavior events, comprising steps as the followings:
(1) acquiring classroom teaching video data to be trained;
(2) dividing the classroom teaching video data to be trained into multiple events according to utterances of a teacher by using a voice activity detection technology; and
(3) performing multi-modal recognition on all events by using multiple artificial intelligence technologies to divide the events into sub-events in multiple dimensions, establishing an event descriptive model according to the sub-events, and describing various teaching behavior events of the teacher in a classroom;
wherein step (3) further comprises:
extracting commonality between events, establishing an event descriptive model that uniformly describes all events according to the commonality and the sub-events, and uniformly describing all teaching behavior events of the teacher that occur in the classroom;
wherein in the event descriptive model, an entire classroom teaching event sequence (E) is defined, E={e1, e2, . . . , en}, n indicates that n events occur, ei indicates an event, and ei is expressed by a 6-tuple <id, t, dt, w, aw, R>, wherein id is a unique identifier of an event;
t is a start time of the event;
dt is a duration corresponding to the event whose start time is t;
w is a dimension of the event, w∈W, W={w0, w1, w2, . . . , wm}, and the dimension comprises the teacher's facial expression, speech emotion, gaze, hand gesture, and body posture;
aw is an attribute of an event w, aw∈{a1w, a2w, . . . , alw}R indicates events correlated with a current event and correlations therebetween, and is a 2-tuple sequence defined as R={<e1, r1>, <e2, r2>, . . . , <en, rn>}, where e in a relational 2-tuple <e, r> indicates an event associated with the current event, and r indicates a specific value of the correlation between the two events.