US 11,780,083 B2
Determining and utilizing corrections to robot actions
Nicolas Hudson, San Mateo, CA (US); and Devesh Yamparala, Mountain View, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by GOOGLE LLC, Mountain View, CA (US)
Filed on Nov. 5, 2021, as Appl. No. 17/520,175.
Application 17/520,175 is a continuation of application No. 16/728,159, filed on Dec. 27, 2019, granted, now 11,198,217.
Application 16/728,159 is a continuation of application No. 15/640,914, filed on Jul. 3, 2017, granted, now 10,562,181, issued on Feb. 18, 2020.
Prior Publication US 2022/0055209 A1, Feb. 24, 2022
Int. Cl. B25J 13/08 (2006.01); B25J 9/16 (2006.01); G05B 13/02 (2006.01); G06N 3/084 (2023.01); G06N 3/008 (2023.01); G06N 3/045 (2023.01)
CPC B25J 9/161 (2013.01) [B25J 9/1602 (2013.01); B25J 9/163 (2013.01); B25J 9/1656 (2013.01); B25J 9/1697 (2013.01); G05B 13/027 (2013.01); G06N 3/008 (2013.01); G06N 3/045 (2023.01); G06N 3/084 (2013.01); G05B 2219/33036 (2013.01); G05B 2219/33037 (2013.01)] 17 Claims
OG exemplary drawing
 
1. A method implemented by one or more processors, the method comprising:
receiving, via at least one microphone of a robot, spoken natural language user interface input provided by a user;
processing the spoken natural language user interface input to determine that the spoken natural language user interface input conforms to a particular classification;
determining, based on processing vision sensor data using one or more neural network models locally stored on the robot, that an object in an environment of the robot has the particular classification;
in response to determining that the spoken natural language user interface input conforms to the particular classification and determining that the object has the particular classification:
providing control commands to one or more actuators of the robot to autonomously perform a robotic action directed toward the object, wherein the robotic action comprises autonomously moving one or more components of the robot closer toward the object; and
during autonomous performance of the robotic action directed toward the object, rendering, via at least one user interface output device of the robot, natural language user interface output that includes the particular classification.