US 12,423,821 B2
Systems and methods for interacting with a large language model
Noah Deutsch, San Francisco, CA (US); and Benjamin Zweig, San Francisco, CA (US)
Assigned to OpenAI OPCo, LLC, San Francisco, CA (US)
Filed by OpenAI Opco, LLC, San Francisco, CA (US)
Filed on Jun. 13, 2024, as Appl. No. 18/742,069.
Application 18/742,069 is a continuation of application No. 18/475,722, filed on Sep. 27, 2023, granted, now 12,051,205.
Prior Publication US 2025/0104243 A1, Mar. 27, 2025
Int. Cl. G06T 7/10 (2017.01)
CPC G06T 7/10 (2017.01) [G06T 2200/24 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
receiving a prompt associated with a graphical user interface (GUI);
generating input data using a snapshot of the GUI and the prompt, the input data generated in a format usable by a machine learning model, wherein generating input data comprises:
tokenizing the snapshot of the GUI and the prompt to generate a tokenized snapshot of the GUI and a tokenized prompt; and
concatenating the tokenized snapshot of the GUI and the tokenized prompt into a singular tokenized input;
generating an output by applying the input data to the machine learning model, the machine learning model being configured to identify a location in the GUI based on the prompt, the output comprising a location indication within the GUI; and
generating instructions to display a cursor at the location in the GUI.