US 11,935,530 B2
Multimodal responses
April Pufahl, Mountain View, CA (US); Jared Strawderman, San Jose, CA (US); Harry Yu, San Francisco, CA (US); Adriana Olmos Antillon, San Francisco, CA (US); Jonathan Livni, San Francisco, CA (US); Okan Kolak, Sunnyvale, CA (US); James Giangola, Mountain View, CA (US); Nitin Khandelwal, Sunnyvale, CA (US); Jason Kearns, Oakland, CA (US); Andrew Watson, Zurich (CH); Joseph Ashear, Redwood City, CA (US); and Valerie Nygaard, Saratoga, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Nov. 1, 2021, as Appl. No. 17/515,901.
Application 17/515,901 is a continuation of application No. 16/251,982, filed on Jan. 18, 2019, granted, now 11,164,576.
Claims priority of provisional application 62/726,947, filed on Sep. 4, 2018.
Prior Publication US 2022/0051675 A1, Feb. 17, 2022
Int. Cl. G10L 15/22 (2006.01); G06F 1/16 (2006.01); G06F 3/16 (2006.01)
CPC G10L 15/22 (2013.01) [G06F 1/1694 (2013.01); G06F 3/167 (2013.01); G06F 2203/0381 (2013.01); G10L 2015/223 (2013.01); G10L 2015/225 (2013.01); H04M 2203/253 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A method implemented by one or more processors, the method comprising:
determining a client device action based on one or more instances of user interface input provided by a user of a multimodal client device;
determining, based at least in part on sensor data from one or more sensors of the multimodal client device, a current client device modality of the multimodal client device,
wherein the current client device modality is one of a plurality of discrete client device modalities available for the multimodal client device, and
wherein the sensor data based on which the current client device modality is determined is in addition to any sensor data generated by the one or more instances of user interface input;
generating client device output for the client device action using a multimodal response,
wherein the multimodal response includes:
first output characteristics for the current client device modality and for the client device action, and
second output characteristics for a second client device modality and for the client device action, and
wherein generating the client device output comprises selecting the first output characteristics in response to determining the current client device modality; and
causing the client device output to be rendered by the multimodal client device;
while at least part of the client device output is being rendered by the one or more user interface output devices of the multimodal client device:
detecting, based at least in part on updated sensor data from the one or more sensors, a switch of the multimodal client device from the current client device modality to the second client device modality;
in response to detecting the switch, generating alternate client device output using the multimodal response, wherein generating the alternate client device output comprises selecting the second output characteristics in response to detecting the switch to the second client device modality; and
causing the alternate client device output to be rendered by the multimodal client device.