US 12,113,934 B1
Systems and methods for intelligent call agent evaluations
Bryan Dempsey, Raleigh, NC (US); Saquib Ilahi, Bellingham, MA (US); Jenson Joy, North Attleboro, MA (US); Nirupam Sarkar, Westford, MA (US); Murad Maayah, Morrisville, NC (US); Abigail Parker, Cary, NC (US); Meagan Gilbert, Durham, NC (US); and Derek Kaschl, Cary, NC (US)
Assigned to FMR LLC, Boston, MA (US)
Filed by FMR LLC, Boston, MA (US)
Filed on Apr. 4, 2024, as Appl. No. 18/626,879.
Int. Cl. H04M 3/51 (2006.01); G09B 5/02 (2006.01); G10L 15/18 (2013.01); G10L 15/183 (2013.01); H04M 3/42 (2006.01)
CPC H04M 3/5175 (2013.01) [G09B 5/02 (2013.01); G10L 15/1815 (2013.01); G10L 15/183 (2013.01); H04M 3/42221 (2013.01); H04M 2201/405 (2013.01); H04M 2201/42 (2013.01); H04M 2203/401 (2013.01); H04M 2203/403 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A computer-implemented method for quantitative performance evaluation of a call agent, the method comprising:
receiving, by a computing device, an audio recording of a call between the call agent and a customer;
converting, by the computing device, the audio recording to a text-based transcript;
identifying, by the computing device, at least one topic for categorizing the transcript by comparing the transcript to a list of historical call topics;
retrieving, by the computing device, a set of criteria associated with the at least one topic, wherein each criterion correlates to a set of predefined questions for interrogating the transcript to evaluate the performance of the call agent with respect to the corresponding criterion, and wherein each question captures a sub-criterion under the corresponding criterion;
inputting, by the computing device, the predefined questions and the transcript into a trained large language model (LLM) to obtain scores for respective ones of the predefined questions, wherein each score measures a degree of satisfaction of the performance of the call agent during the call with respect to the sub-criterion captured by the corresponding predefined question;
extracting, by the computing device, evidence from the transcript for each predefined question, wherein the evidence is a part of the transcript from which the LLM derived the score in response to the corresponding predefined question;
comparing the extracted evidence against the transcript to determine a degree of match;
rejecting the score generated for the predefined question if the degree of match is below a predefined threshold;
calculating, by the computing device, a combined score for the call agent based on the scores for the predefined questions that are not rejected, the combined score representing an overall degree of satisfaction of the performance of the call agent with respect to the at least one topic identified for the call; and
presenting, by the computing device, via a graphical user interface, an identification of the call agent, a link to the audio recording of the call, the topic identified, and the combined score that quantitatively evaluates the performance of the call agent during the call.