US 12,444,410 B2
Gestural prompting based on conversational artificial intelligence
Abhishek Rohatgi, Roxboro (CA); Eduardo Olvera, Phoenix, AZ (US); Dinesh Samtani, Mississaug (CA); Flaviu Gelu Negrean, Montreal (CA); and Manar Alazma, Lexington, MA (US)
Assigned to Microsoft Technology Licensing, LLC., Redmond, WA (US)
Filed by NUANCE COMMUNICATIONS, INC., Burlington, MA (US)
Filed on Jul. 26, 2022, as Appl. No. 17/874,146.
Prior Publication US 2024/0038225 A1, Feb. 1, 2024
Int. Cl. G10L 15/18 (2013.01); G06F 3/01 (2006.01); G06F 40/30 (2020.01); G06F 40/35 (2020.01); G06N 3/08 (2023.01); G06N 20/00 (2019.01); G10L 15/06 (2013.01); G10L 15/22 (2006.01); G10L 15/24 (2013.01); G10L 15/30 (2013.01)
CPC G10L 15/1815 (2013.01) [G06F 3/017 (2013.01); G06F 40/30 (2020.01); G06F 40/35 (2020.01); G06N 3/08 (2013.01); G06N 20/00 (2019.01); G10L 15/063 (2013.01); G10L 15/22 (2013.01); G10L 15/24 (2013.01); G10L 15/30 (2013.01); G10L 2015/227 (2013.01); G10L 2015/228 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
obtaining training data that describes (a) a situation, (b) a gesture for a response to the situation, (c) a prompt to accompany the response, and (d) a gestural annotation for the response, the training data including a hypothetical user query related to a custom domain,
the situation being a stimulus warranting a response from a natural language understanding (NLU) model,
the gesture for the response being a suggested gesture in response to the situation from the NLU model,
the prompt to accompany the response being a suggested phrase in response to the situation from the NLU model, and
the gestural annotation for the response being a process of labeling the response to show a gestural outcome to be predicted by the NLU model;
the training data comprising an utterance, an intent, an entity, a vocabulary, a gesture, or an action usable to respond to the situation;
tagging the hypothetical user query with the gestural annotation for the response; and
training the NLU model to address the situation, based on the training data, the training comprising:
based on the tagged hypothetical user query, training the NLU model to recognize a plurality of gesture types, and
based on recognizing a gesture type of the plurality of gesture types, train the NLU model to classify and annotate a real user query at a runtime based on the recognized gesture type.