US 12,112,138 B2
Systems and methods for an end-to-end evaluation and testing framework for task-oriented dialog systems
Guangsen Wang, Singapore (SG); Samson Min Rong Tan, Singapore (SG); Shafiq Rayhan Joty, Singapore (SG); Gang Wu, Santa Clara, CA (US); Chu Hong Hoi, Singapore (SG); and Ka Chun Au, San Francisco, CA (US)
Assigned to Salesforce, Inc., San Francisco, CA (US)
Filed by Salesforce, Inc., San Francisco, CA (US)
Filed on Jun. 2, 2022, as Appl. No. 17/830,889.
Claims priority of provisional application 63/303,850, filed on Jan. 27, 2022.
Prior Publication US 2023/0237275 A1, Jul. 27, 2023
Int. Cl. G06F 40/35 (2020.01); G06F 40/186 (2020.01); G06F 40/40 (2020.01); H04L 51/02 (2022.01)
CPC G06F 40/35 (2020.01) [G06F 40/186 (2020.01); G06F 40/40 (2020.01); H04L 51/02 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of simulating a dialog, the method comprising:
receiving, via a communication interface, a plurality of task-oriented dialog data generated from a dialog agent;
determining a plurality of natural language understanding pairs including bot dialog acts and respective bot messages based on the plurality of task-oriented dialog data;
determining a plurality of goal pairs including goal entity slots and respective goal entity slot values based on the plurality of task-oriented dialog data;
determining a plurality of natural language generation templates based on the plurality of task-oriented dialog data;
generating a simulated task-oriented dialog based on the plurality of natural language understanding pairs, the plurality of goal pairs, and the plurality of natural language generation templates;
generating simulation results from the simulated task-oriented dialog; and
generating, based on simulation results, an actionable suggestion relating to an adjustment to the dialog agent.