US 12,145,266 B2
Robot systems, methods, control modules, and computer program products that leverage large language models
Geordie Rose, Vancouver (CA); and Suzanne Gildert, Vancouver (CA)
Assigned to Sanctuary Cognitive Systems Corporation, Vancouver (CA)
Filed by Sanctuary Cognitive Systems Corporation, Vancouver (CA)
Filed on Jan. 19, 2024, as Appl. No. 18/417,081.
Claims priority of provisional application 63/441,897, filed on Jan. 30, 2023.
Prior Publication US 2024/0253224 A1, Aug. 1, 2024
Int. Cl. B25J 9/16 (2006.01); B25J 19/02 (2006.01); G06F 40/279 (2020.01); G06F 40/40 (2020.01)
CPC B25J 9/1602 (2013.01) [B25J 9/161 (2013.01); B25J 9/163 (2013.01); B25J 9/1653 (2013.01); B25J 9/1658 (2013.01); B25J 9/1661 (2013.01); B25J 9/1671 (2013.01); B25J 9/1697 (2013.01); B25J 19/02 (2013.01); B25J 19/023 (2013.01); G06F 40/279 (2020.01); G06F 40/40 (2020.01)] 17 Claims
OG exemplary drawing
 
1. A method of operation of a robot system to perform a work objective, wherein the robot system includes a robot body, the method comprising:
for an initial task plan that successfully completes a first instance of the work objective:
parameterizing the initial task plan, wherein parameterizing the initial task plan includes replacing at least one noun in the initial task plan with a variable to be populated in a subsequent invocation of the parameterized task plan; and
storing the parameterized task plan;
capturing, by at least one sensor of the robot system, sensor data representing information about an environment of the robot body;
generating, by at least one processor of the robot system, a natural language (NL) description of at least one aspect of the environment based at least in part on the sensor data, the NL description of at least one aspect of the environment including at least one noun corresponding to at least one object in the environment;
providing a NL query to a large language model (LLM) module, the NL query including the NL description of at least one aspect of the environment, an NL description of a second instance of the work objective, an NL description of an instruction set executable by the robot system, and an NL request for a task plan;
accessing the parameterized task plan based on the NL description of the second instance of the work objective;
populating the parameterized task plan with at least one noun from the NL description of at least one aspect of the environment, wherein populating the parameterized task plan with at least one noun from the NL description of the environment includes replacing each variable in the parameterized task plan with a corresponding noun from the NL description of the environment;
receiving the populated task plan from the LLM module, the populated task plan expressed in NL; and
executing the populated task plan by the robot system to perform the second instance of the work objective.