US 12,236,064 B2
Encoding/decoding user interface interactions
Keun Soo Yim, San Jose, CA (US); and Ilya Firman, Sunnyvale, CA (US)
Assigned to Google LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Jul. 5, 2023, as Appl. No. 18/347,374.
Application 18/347,374 is a continuation of application No. 17/651,057, filed on Feb. 14, 2022, granted, now 11,726,641.
Prior Publication US 2023/0350550 A1, Nov. 2, 2023
Int. Cl. G06F 3/0484 (2022.01); G06F 3/0481 (2022.01); G06F 3/16 (2006.01); G06F 9/451 (2018.01); G06F 40/205 (2020.01); G06F 40/30 (2020.01); G06V 20/70 (2022.01); G10L 15/18 (2013.01); G10L 15/22 (2006.01)
CPC G06F 3/0484 (2013.01) [G06F 3/0481 (2013.01); G06F 3/167 (2013.01); G06F 9/451 (2018.02); G06F 40/205 (2020.01); G06F 40/30 (2020.01); G06V 20/70 (2022.01); G10L 15/1815 (2013.01); G10L 15/22 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
obtaining a query directed toward a digital assistant application executing on a user device, the query specifying a user intent for interacting with a target application not currently executing on the user device;
obtaining an action for the target application, the action representing a high-level description of the user intent specified by the query for interacting with the target application;
selecting an application-specific script from a plurality of application-specific scripts based on the target application and the action, each application-specific script of the plurality of application-specific scripts configured to cause a corresponding application to perform the action;
based on the selected application-specific script, translating the action for the target application into a translated application-specific script, the translated application-specific script converting the high-level description of the user intent into an application-specific user interface interaction command for interacting with a user interface of the target application; and
executing the translated application-specific script to fulfill the user intent specified by the query, wherein execution of the translated application-specific script causes the target application to input text in the user interface of the target application.