US 11,701,090 B2
Ultrasound bone registration with learning-based segmentation and sound speed calibration
José Luis Moctezuma de la Barrera, Freiburg (DE); Mehrdad Salehi, Munich (DE); Raphael Prevost, Munich (DE); and Wolfgang Wein, Munich (DE)
Assigned to MAKO Surgical Corp., Weston, FL (US)
Filed by MAKO Surgical Corp., Ft. Lauderdale, FL (US)
Filed on Aug. 16, 2018, as Appl. No. 15/999,152.
Claims priority of provisional application 62/546,158, filed on Aug. 16, 2017.
Prior Publication US 2019/0069882 A1, Mar. 7, 2019
Int. Cl. A61B 8/08 (2006.01); A61B 6/00 (2006.01); A61B 34/10 (2016.01); A61B 8/00 (2006.01); G06T 7/33 (2017.01); G06T 7/11 (2017.01); A61B 6/03 (2006.01)
CPC A61B 8/5207 (2013.01) [A61B 6/5247 (2013.01); A61B 8/0875 (2013.01); A61B 8/4416 (2013.01); A61B 8/5223 (2013.01); A61B 8/5238 (2013.01); A61B 34/10 (2016.02); G06T 7/33 (2017.01); A61B 6/032 (2013.01); A61B 8/58 (2013.01); A61B 2034/105 (2016.02); G06T 7/11 (2017.01); G06T 2207/10132 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A method of operating a surgical system, the surgical system comprising an imaging device, a robotic manipulator configured to move a surgical instrument to manipulate an anatomical object with a working end of the surgical instrument, and one or more controllers, the method comprising:
generating, with the imaging device, ultrasound imaging of the anatomical object by propagating ultrasound waves along a plurality of scanlines through the anatomical object;
segmenting, with the one or more controllers, the ultrasound imaging with a convolutional neural network to detect a surface of the anatomical object by generating a probability map with the convolutional neural network and extracting the surface from the probability map for each of the plurality of scanlines;
calibrating, with the one or more controllers, the ultrasound imaging to reflect a variation in propagation speed of ultrasound waves through the anatomical object by comparing first and second steered frames of the ultrasound imaging with a third frame of the ultrasound imaging that is angled between the first and second steered frames, and wherein calibrating further comprises computing a first difference between the first steered frame and the third frame, computing a second difference between the second steered frame and the third frame, computing a sum of the first and second differences, and optimizing the sum to minimize a cost function;
temporally calibrating, with the one or more controllers, the ultrasound imaging with respect to a tracking coordinate system by creating a point cloud of the surface, calculating a set of projection values of the point cloud to a vector oriented relative to the plurality of scanlines, and calculating a temporal lag for the ultrasound imaging by minimizing variance of the set of projection values;
obtaining, with the one or more controllers, co-modality imaging of the anatomical object and a predefined boundary associated with the anatomical object;
automatically registering, with the one or more controllers, the segmented and calibrated ultrasound imaging to the co-modality imaging; and
after automatically registering, controlling, with the one or more controllers, the robotic manipulator for moving the surgical instrument to manipulate the anatomical object while preventing the working end of the surgical instrument from extending beyond the predefined boundary.