US 11,941,734 B2
Rendering tool information as graphic overlays on displayed images of tools
Brandon D. Itkowitz, San Jose, CA (US); Simon P. DiMaio, San Carlos, CA (US); Daniel J. Halabe, Los Altos, CA (US); Christopher J. Hasser, Los Altos, CA (US); Brian David Hoffman, Mountain View, CA (US); David Q. Larkin, Menlo Park, CA (US); Catherine J. Mohr, Mountain View, CA (US); Paul W. Mohr, Mountain View, CA (US); Tao Zhao, Sunnyvale, CA (US); and Wenyi Zhao, Weston, FL (US)
Assigned to INTUITIVE SURGICAL OPERATIONS, INC., Sunnyvale, CA (US)
Filed by Intuitive Surgical Operations, Inc., Sunnyvale, CA (US)
Filed on Mar. 26, 2021, as Appl. No. 17/213,924.
Application 17/213,924 is a continuation of application No. 16/351,444, filed on Mar. 12, 2019, granted, now 10,984,567.
Application 16/351,444 is a continuation of application No. 15/946,408, filed on Apr. 5, 2018, granted, now 10,282,881, issued on May 7, 2019.
Application 15/946,408 is a continuation of application No. 13/768,047, filed on Feb. 15, 2013, granted, now 10,008,017, issued on Jun. 26, 2018.
Application 13/768,047 is a continuation in part of application No. 12/415,354, filed on Mar. 31, 2009, granted, now 9,789,608, issued on Oct. 17, 2017.
Prior Publication US 2021/0256749 A1, Aug. 19, 2021
Int. Cl. G06T 11/60 (2006.01); A61B 34/30 (2016.01); A61B 34/37 (2016.01); A61B 90/00 (2016.01); B25J 9/16 (2006.01); G05B 19/42 (2006.01); G06T 11/40 (2006.01); A61B 34/00 (2016.01); A61B 34/20 (2016.01)
CPC G06T 11/60 (2013.01) [A61B 34/30 (2016.02); A61B 34/37 (2016.02); A61B 90/36 (2016.02); B25J 9/1666 (2013.01); B25J 9/1671 (2013.01); B25J 9/1692 (2013.01); G05B 19/4202 (2013.01); G06T 11/40 (2013.01); A61B 34/20 (2016.02); A61B 2034/2061 (2016.02); A61B 34/25 (2016.02); A61B 90/361 (2016.02); A61B 2090/371 (2016.02); G05B 2219/36432 (2013.01); G05B 2219/39083 (2013.01); G05B 2219/39096 (2013.01); G05B 2219/39449 (2013.01); G05B 2219/40607 (2013.01); G05B 2219/45117 (2013.01); G05B 2219/45123 (2013.01); Y02P 90/02 (2015.11)] 18 Claims
OG exemplary drawing
 
1. A robotic system comprising:
a first robotic arm operatively coupleable to a first tool, the first tool having a first working end;
an image capture device;
a display; and
a processor configured to:
cause an image of a work site, which was captured by the image capture device from a perspective of an image reference frame, to be displayed on the display, the image of the work site including an image of the first working end of the first tool;
determine a position of the first working end of the first tool in the image of the work site;
render a tool information overlay at the position of the first working end of the first tool in the image of the work site, wherein the tool information overlay visually indicates state information for the first tool; and
change the tool information overlay while the first tool is in a first operational state, wherein changing the tool information overlay includes changing a brightness of the tool information overlay.