US 11,938,638 B2
Simulation driven robotic control of real robot(s)
Yunfei Bai, Fremont, CA (US); Tigran Gasparian, Munich (DE); Brent Austin, Munich (DE); Andreas Christiansen, Lengdorf (DE); Matthew Bennice, San Jose, CA (US); and Paul Bechard, San Jose, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by GOOGLE LLC, Mountain View, CA (US)
Filed on Jun. 3, 2021, as Appl. No. 17/337,815.
Claims priority of provisional application 63/132,582, filed on Dec. 31, 2020.
Prior Publication US 2022/0203535 A1, Jun. 30, 2022
Int. Cl. B25J 9/16 (2006.01)
CPC B25J 9/1671 (2013.01) [B25J 9/1661 (2013.01); B25J 9/1697 (2013.01)] 21 Claims
OG exemplary drawing
 
1. A method implemented by one or more processors, the method comprising:
configuring a simulated environment, of a robotic simulator, based on real environment state data that is based on one or more sensor-based observations of a real environment;
determining, using the robotic simulator, a sequence of robotic actions for use in controlling a real robot, in the real environment, in performing at least part of a robotic task;
applying, to a simulated robot of the robotic simulator, the sequence of robotic actions to generate a sequence of simulated state data instances, wherein each of the simulated state data instances defines, for a corresponding point in the sequence of robotic actions one or both of:
corresponding anticipated environmental state data for the corresponding point, and
corresponding anticipated robot state data for the corresponding point;
subsequent to determining the sequence of robotic actions:
controlling the real robot to implement a first subset of the sequence of robotic actions;
comparing, during or after controlling the real robot to implement the first subset of the sequence of robotic actions, a real state data instance to a corresponding one of the simulated state data instances for the corresponding point in the sequence of robotic actions, the real state data instance defining, for the corresponding point, one or both of:
updated real environment state data, that is based on one or more updated sensor-based observations of the real environment, and
real robot state data that is based on one or more sensor-based observations of the real robot; and
in response to the comparing indicating a threshold degree of similarity between the updated real state data and the corresponding one of the anticipated state data instances for the corresponding point:
controlling the real robot to implement a second subset of the sequence of robotic actions,
wherein controlling the real robot to fully implement the second subset of the sequence of robotic actions is contingent on the comparing indicating the threshold degree of similarity between the updated real state data and the corresponding one of the anticipated state data instances for the corresponding point.