US 12,134,199 B2
Determining environment-conditioned action sequences for robotic tasks
Soeren Pirk, Palo Alto, CA (US); Seyed Mohammad Khansari Zadeh, San Carlos, CA (US); Karol Hausman, San Francisco, CA (US); and Alexander Toshev, San Francisco, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Appl. No. 17/642,325
Filed by GOOGLE LLC, Mountain View, CA (US)
PCT Filed Sep. 9, 2020, PCT No. PCT/US2020/049851
§ 371(c)(1), (2) Date Mar. 11, 2022,
PCT Pub. No. WO2021/050488, PCT Pub. Date Mar. 18, 2021.
Claims priority of provisional application 62/900,603, filed on Sep. 15, 2019.
Prior Publication US 2022/0331962 A1, Oct. 20, 2022
Int. Cl. B25J 9/16 (2006.01); G06N 3/045 (2023.01); G06V 10/147 (2022.01); G06V 10/44 (2022.01); G06V 10/82 (2022.01); G06V 20/10 (2022.01); G06V 20/17 (2022.01); G06V 20/13 (2022.01)
CPC B25J 9/1697 (2013.01) [B25J 9/161 (2013.01); B25J 9/163 (2013.01); B25J 9/1664 (2013.01); B25J 9/1669 (2013.01); B25J 9/1679 (2013.01); G06N 3/045 (2023.01); G06V 10/147 (2022.01); G06V 10/454 (2022.01); G06V 10/82 (2022.01); G06V 20/10 (2022.01); G06V 20/17 (2022.01); G06V 20/13 (2022.01)] 14 Claims
OG exemplary drawing
 
1. A method implemented by one or more processors of a robot, the method comprising:
processing an instance of sensor data using an environment-conditioned action sequence prediction model, wherein the sensor data includes an instance of vision data captured by a vision component of the robot, wherein the vision data comprises an image of at least one object in the environment of the robot, and wherein the environment-conditioned action sequence prediction model is a trained machine learning model;
determining, based on output generated based on the processing using the environment-conditioned action sequence prediction model, a first set of predicted actions for an object manipulation robotic task associated with the at least one object, and a particular order for performing the predicted actions of the set;
controlling the robot to perform the predicted actions of the first set in the particular order, wherein controlling the robot to perform each of the predicted actions of the first set in the particular order comprises:
for each of the predicted actions, and in the particular order:
selecting a corresponding action network that corresponds to the predicted action;
until determining that the predicted action is complete:
processing corresponding additional instances of sensor data, of the robot, using the corresponding action network, and
controlling the robot based on action output, generated based on the processing using the corresponding action network.