US 12,334,051 B2
Computationally reacting to a multiparty conversation
Andrew Reece, San Francisco, CA (US); Peter Bull, San Francisco, CA (US); Gus Cooney, San Francisco, CA (US); Casey Fitzpatrick, San Francisco, CA (US); Gabriella Rosen Kellerman, San Francisco, CA (US); and Ryan Sonnek, New Hope, MN (US)
Assigned to BetterUp, Inc., Austin, TX (US)
Filed by BetterUp, Inc., San Francisco, CA (US)
Filed on Jul. 11, 2022, as Appl. No. 17/811,867.
Application 17/811,867 is a continuation of application No. 16/798,248, filed on Feb. 21, 2020, granted, now 11,417,318.
Prior Publication US 2022/0343899 A1, Oct. 27, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G10L 15/22 (2006.01); G06N 20/00 (2019.01); G06V 20/40 (2022.01); G10L 15/06 (2013.01)
CPC G10L 15/063 (2013.01) [G06N 20/00 (2019.01); G06V 20/49 (2022.01); G10L 15/22 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for computationally reacting to conversations, the method comprising:
receiving one or more multiparty videos representing a conversation between at least a first user and a second user, each multiparty video including acoustic data and video data; and
generating conversation features based on the one or more multiparty videos by:
segmenting the one or more multiparty videos into multiple utterances;
identifying, for each the multiple utterances, data for multiple modalities; and
extracting conversation features, for each particular utterance of the multiple utterances, from each of the data for the multiple modalities associated with that particular utterance;
generating conversation analysis indicators, comprising one or more conversation scores for the conversation, by applying the conversation features to a plurality of machine learning systems, wherein generating the conversation analysis indicators comprises receiving the one or more conversation scores from the plurality of machine learning systems, wherein each machine learning system is individually trained using labeled data sets for a different conversation analysis indicator;
applying a mapping of the conversation analysis indicators to inferences or actions to determine inference or action results;
reacting to the conversation according to the inference or action results mapped to the conversation analysis indicators in the mapping;
transmitting a first alert notification to a user device associated with the first user if a progress score is above a first predetermined threshold value, the progress score being determined based on profile of the first user and the conversation analysis indicators; and
transmitting a second alert notification to a coaching device associated with the second user if the conversation analysis indicators are below a second predetermined threshold value.