US 11,941,719 B2
Learning robotic tasks using one or more neural networks
Jonathan Tremblay, Redmond, WA (US); Stan Birchfield, Sammamish, WA (US); Stephen Tyree, University City, MO (US); Thang To, Redmond, WA (US); Jan Kautz, Lexington, MA (US); and Artem Molchanov, Bellevue, WA (US)
Assigned to NVIDIA Corporation, Santa Clara, CA (US)
Filed by Nvidia Corporation, Santa Clara, CA (US)
Filed on Jan. 23, 2019, as Appl. No. 16/255,038.
Claims priority of provisional application 62/620,938, filed on Jan. 23, 2018.
Prior Publication US 2019/0228495 A1, Jul. 25, 2019
Int. Cl. G06T 9/00 (2006.01); B25J 9/16 (2006.01); G05B 13/00 (2006.01); G06N 3/08 (2023.01); G06T 1/00 (2006.01); G06T 7/73 (2017.01); G05D 1/00 (2006.01)
CPC G06T 1/0014 (2013.01) [B25J 9/161 (2013.01); B25J 9/1661 (2013.01); B25J 9/1697 (2013.01); G05B 13/00 (2013.01); G06N 3/08 (2013.01); G06T 7/73 (2017.01); G05D 1/0088 (2013.01); G05D 1/0221 (2013.01); G05D 2201/0213 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method, comprising:
receiving image data representative of a task being physically performed;
inferring, using the image data as input to a perception neural network, a relationship between at least two objects resulting from performance of the task;
inferring, using the relationship as input to a plan generation neural network, a plan corresponding to the relationship between the at least two objects, the plan providing at least a human-readable representation of the task;
receiving confirmation of the plan; and
inferring, using an execution neural network and the plan, an instruction readable by a robotic device to cause the robotic device, upon execution of the instruction, to perform the task.