US 12,189,870 B2
Learned computer control using pointing device and keyboard actions
Peter Conway Humphreys, London (GB); Timothy Paul Lillicrap, London (GB); Tobias Markus Pohlen, London (GB); and Adam Anthony Santoro, London (GB)
Assigned to Deep Mind Technologies Limited, London (GB)
Filed by DeepMind Technologies Limited, London (GB)
Filed on Jan. 30, 2023, as Appl. No. 18/103,309.
Claims priority of provisional application 63/304,549, filed on Jan. 28, 2022.
Prior Publication US 2023/0244325 A1, Aug. 3, 2023
Int. Cl. G06F 3/023 (2006.01); G06F 3/033 (2013.01); G06F 40/284 (2020.01)
CPC G06F 3/033 (2013.01) [G06F 3/023 (2013.01); G06F 40/284 (2020.01)] 32 Claims
OG exemplary drawing
 
1. A computer-implemented method for controlling a particular computer to execute a task, the method comprising:
receiving a control input comprising a visual input, the visual input comprising one or more screen frames of a computer display that represent at least a current state of the particular computer;
processing the control input using a neural network to generate a plurality of control outputs that are used to control the particular computer to execute the task, wherein the plurality of control outputs comprise (i) an action type output that specifies a single discrete probability distribution over a set of possible actions that could be performed to control the computer, wherein the set of possible actions comprise at least a pointing device action and a keyboard action and (ii) a cursor coordinate output that specifies two discrete probability distributions over possible cursor coordinates;
determining one or more actions based on at least one of the single discrete probability distribution specified by the action type output and the two discrete probability distributions specified by the cursor coordinate output; and
executing the one or more actions to control the particular computer.