US 11,900,960 B2
System and method for frustration detection
Jessica Perri, Salt Lake City, UT (US); Amelie Stephan, Cupertino, CA (US); Julia Laski, Aurora, IL (US); Mark Schmelzenbach, Layton, UT (US); Sara Olson, Salt Lake City, UT (US); and Shaun Matthews, Salt Lake City, UT (US)
Assigned to Nice Ltd., Ra'anana (IL)
Filed by NICE Ltd., Ra'anana (IL)
Filed on Feb. 17, 2022, as Appl. No. 17/674,350.
Application 17/674,350 is a continuation of application No. 16/776,601, filed on Jan. 30, 2020, granted, now 11,302,346.
Claims priority of provisional application 62/816,459, filed on Mar. 11, 2019.
Prior Publication US 2022/0172738 A1, Jun. 2, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G10L 25/63 (2013.01); G06F 16/683 (2019.01); G10L 15/06 (2013.01); G10L 15/183 (2013.01); G10L 15/22 (2006.01); G06N 20/00 (2019.01)
CPC G10L 25/63 (2013.01) [G10L 15/063 (2013.01); G10L 15/183 (2013.01); G10L 15/22 (2013.01); G06N 20/00 (2019.01)] 17 Claims
OG exemplary drawing
 
1. A method for automatically detecting frustration in an interaction, the method comprising:
identifying in the interaction, by a computer processor, using a set of linguistic rules, natural language patterns related to frustration, wherein the linguistic rules define weights associated with the natural language patterns and rule metadata;
reviewing, by the computer processor, the rule metadata associated with the identified natural language patterns to identify override attributes, wherein an override attribute is selected from a list consisting of: set interaction to frustrated, and set interaction to non-frustrated, wherein:
if the rule metadata does not include override attributes, then a frustration level in the interaction is determined, by the computer processor, based on the identified natural language patterns and weights associated with the identified natural language patterns;
if the rule metadata includes a set interaction to non- frustrated override attribute, then the frustration level is determined to be non-frustrated, and
if the rule metadata includes a set interaction to frustrated override attribute, then the frustration level is determined to be frustrated; and
generating the set of linguistic rules by:
obtaining a plurality of labeled interactions, wherein a label indicates a known level of frustration in each of the labeled interactions;
obtaining identification of phrases associated with frustration that are present in the labeled interactions that are labeled as including frustration;
obtaining a linguistic rule for each of the identified phrases;
training a classifier to calculate a weight for each of the linguistic rules;
performing error analysis to evaluate performance of each linguistic rule;
adjusting the weights based on the error analysis; and
associating metadata to each linguistic rule.