US 11,989,502 B2
Implicitly annotating textual data in conversational messaging
David Yi Xiao, Boston, MA (US); Smit Anish Kiri, Cambridge, MA (US); Tianxing Liu, Waltham, MA (US); Casey Koppes, Hermosa Beach, CA (US); Gabriel Gralla, Somerville, MA (US); Kaila Corrington, Cambridge, MA (US); Nithin Gangadharan, Boston, MA (US); Prisca Sara Joseph, Canton, MA (US); Robert Roosevelt Mercer, III, Raleigh, NC (US); Vera Guttenberger, Cambridge, MA (US); and Andrew Cole Young, Young, MA (US)
Assigned to Klaviyo, Inc, Boston, MA (US)
Filed by Klaviyo, Inc, Boston, MA (US)
Filed on Jan. 31, 2023, as Appl. No. 18/103,571.
Claims priority of provisional application 63/353,569, filed on Jun. 18, 2022.
Prior Publication US 2023/0409817 A1, Dec. 21, 2023
Int. Cl. G06Q 30/02 (2023.01); G06F 3/0482 (2013.01); G06F 40/169 (2020.01)
CPC G06F 40/169 (2020.01) [G06F 3/0482 (2013.01); G06Q 30/0281 (2013.01)] 17 Claims
OG exemplary drawing
 
1. A method, comprising:
receiving, by a server, a first input text message from a user (customer);
displaying, by the server, the first input text message to an agent (merchant);
displaying, by the server, a configurable menu of responses to the first input text message to the agent (merchant), wherein a natural language intent classification model generates at least one response of the configurable menu of responses;
receiving, by the server, a selection of one of the configurable menu of responses from the agent (merchant);
facilitating, by the server, sending of the selected of one of the configurable menu of responses to the user (customer);
associating and recording the selected one of the configurable menu of responses with the first input text message;
assigning an ID to the selected one of the configurable menu of responses;
generating a plurality of annotated datasets based at least on associating the first input text message with the selected one of the configurable menu of responses having the assigned ID, wherein generating the annotated datasets include labeling data with relevant tags for training the natural language intent classification model;
training the natural language intent classification model based on the plurality of annotated datasets;
receiving, by the server, editing of the selected of one of the configurable menu of responses by the agent;
selecting a change threshold of the edited selected of one of the configurable responses based on a type of conversation classification of the agent, wherein the type of classification includes at least a customer service type or a sale type;
checking, by the server, whether the edited selected of one of the configurable responses has changed greater than the change threshold, comprising checking an edit distance between the selected one of the configurable menu of responses and the edited selected one of the configurable menu of responses; and
maintaining the ID of the edited selected of one of the configurable responses when changed less than the change threshold.