US 12,030,183 B1
Robot systems, methods, control modules, and computer program products that leverage large language models
Geordie Rose, Vancouver (CA); and Suzanne Gildert, Vancouver (CA)
Assigned to Sanctuary Cognitive Systems Corporation, Vancouver (CA)
Filed by Sanctuary Cognitive Systems Corporation, Vancouver (CA)
Filed on Jan. 19, 2024, as Appl. No. 18/417,075.
Claims priority of provisional application 63/441,897, filed on Jan. 30, 2023.
Int. Cl. B25J 9/16 (2006.01)
CPC B25J 9/163 (2013.01) [B25J 9/161 (2013.01); B25J 9/1653 (2013.01); B25J 9/1671 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of operating a robot system including a robot body, the method comprising:
capturing, by at least one sensor of the robot system, sensor data representing information about an environment of the robot body;
generating, by at least one processor of the robot system, a natural language (NL) description of at least one aspect of the environment based on the sensor data;
providing a first NL query to a large language model (LLM) module, the first NL query including the NL description of at least one aspect of the environment, an NL description of a work objective, an NL description of an instruction set executable by the robot system, and an NL request for a task plan;
receiving the task plan from the LLM module, the task plan expressed in NL and including a plurality of steps;
determining, by the at least one processor, whether the task plan includes at least one fault;
if the task plan includes at least one fault:
generating, by the at least one processor, an NL description of the at least one fault;
providing a second NL query to the LLM module, the second NL query including at least the NL description of the at least one fault and an NL request for a resolution plan;
receiving the resolution plan from the LLM module, the resolution plan expressed in NL; and
executing an updated task plan by the robot system, the updated task plan based at least in part on the resolution plan; and
if the task plan does not include at least one fault:
executing the task plan to completion by the robot system.