US 12,277,396 B2
Assessing and improving the deployment of large language models in specific domains
Adam Earle, San Francisco, CA (US); and Ali Ziaei, Redwood City, CA (US)
Assigned to Tenyx, Inc., Palo Alto, CA (US)
Filed by Tenyx, Inc, Palo Alto, CA (US)
Filed on Sep. 30, 2022, as Appl. No. 17/937,383.
Prior Publication US 2024/0111960 A1, Apr. 4, 2024
Int. Cl. G06F 40/35 (2020.01)
CPC G06F 40/35 (2020.01) 14 Claims
OG exemplary drawing
 
1. A method comprising:
receiving, by a customer large language model (LLM), an initialization customer prompt including customer state information associated with a set of customer state information categories;
receiving, by an agent LLM, an initialization agent prompt including agent state information associated with a set of agent state information categories;
generating a synthetic chat comprising chat responses accumulated over a number of iterations between chat responses determined by the customer LLM and chat responses determined by the agent LLM, wherein during each iteration:
the customer LLM determines a customer chat response as one of the chat responses based on a received agent chat response and the initialization customer prompt, or
the agent LLM determines an agent chat response as one of the chat responses based on a received customer chat response and the initialization agent prompt;
extracting, by a summarizer LLM, a summary of the synthetic chat using a summarizer prompt including the set of customer state information categories and the set of agent state information categories, wherein the summary includes state information of the synthetic chat;
scoring, by a scorer module, the synthetic chat by comparing the state information from the summary of the synthetic chat to the customer state information of the initialization customer prompt and the agent state information of the initialization customer prompt;
adjusting, based on a score of the synthetic chat determined by the scorer module, the initialization agent prompt;
receiving, from a user of a user computing device, a first audio message comprising user speech;
converting the user speech into natural language text;
computing, by the agent LLM, a chat response based on the adjusted initialization agent prompt and the natural language text;
converting the chat response into a second audio message comprising a synthetic voice; and
causing communication of the second audio message comprising the synthetic voice to the user of the user computing device.