US 11,861,263 B1
Robust natural language based control of computer applications
Thomas Hunt, Oakland, CA (US); David Andre, San Francisco, CA (US); Nisarg Vyas, Gujarat (IN); Rebecca Radkoff, San Francisco, CA (US); and Rishabh Singh, San Jose, CA (US)
Assigned to X DEVELOPMENT LLC, Mountain View, CA (US)
Filed by X Development LLC, Mountain View, CA (US)
Filed on Jun. 22, 2022, as Appl. No. 17/846,351.
Claims priority of provisional application 63/236,458, filed on Aug. 24, 2021.
Int. Cl. G06F 3/0481 (2022.01); G06F 3/16 (2006.01); G06F 3/0484 (2022.01); G10L 15/22 (2006.01)
CPC G06F 3/167 (2013.01) [G06F 3/0481 (2013.01); G06F 3/0484 (2013.01); G10L 15/22 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method implemented by one or more processors, the method comprising:
receiving natural language input that includes a request to control a computer application;
generating a request embedding, generating the request embedding being based on processing the natural language input;
processing the request embedding to generate a corresponding measure for each of multiple actions, each of the actions being specific to a domain of the computer application and being implementable through interaction with the computer application;
generating, based on the corresponding measures, multiple candidate action sets for the request, each of the candidate action sets including a respective one of the actions or a respective ordered sequence of multiple of the actions, and each of the action sets being unique relative to all other of the action sets;
for each of the candidate action sets for the request, and prior to any interacting, responsive to the request, with the computer application to control the computer application:
performing a simulation of controlling the computer application using the candidate action set, and
generating, based on simulated data from the simulation, a suitability metric for the candidate action set;
prior to any interacting, responsive to the request, with the computer application to control the computer application, automatically selecting, from the candidate action sets and based on the suitability metrics, a given action set for utilization responsive to the request; and
interacting with the computer application to control the computer application using the selected given action set.