US 12,245,741 B2
Methods, systems, and computer readable media for generating and providing artificial intelligence assisted surgical guidance
Vivek Paresh Buch, Philadelphia, PA (US); Peter John Madsen, Philadelphia, PA (US); and Jianbo Shi, Philadelphia, PA (US)
Assigned to THE TRUSTEES OF THE UNIVERSITY OF PENNSYLVANIA, Philadelphia, PA (US)
Appl. No. 17/263,025
Filed by The Trustees of the University of Pennsylvania, Philadelphia, PA (US)
PCT Filed Jul. 25, 2019, PCT No. PCT/US2019/043428
§ 371(c)(1), (2) Date Jan. 25, 2021,
PCT Pub. No. WO2020/023740, PCT Pub. Date Jan. 30, 2020.
Claims priority of provisional application 62/703,400, filed on Jul. 25, 2018.
Prior Publication US 2021/0307841 A1, Oct. 7, 2021
Int. Cl. A61B 1/00 (2006.01); A61B 5/00 (2006.01); A61B 34/20 (2016.01); G06V 10/764 (2022.01); G06V 10/82 (2022.01); G06V 20/20 (2022.01); A61B 90/00 (2016.01)
CPC A61B 1/000096 (2022.02) [A61B 5/489 (2013.01); A61B 5/4893 (2013.01); A61B 5/7267 (2013.01); A61B 5/7275 (2013.01); A61B 5/749 (2013.01); A61B 34/20 (2016.02); G06V 10/764 (2022.01); G06V 10/82 (2022.01); G06V 20/20 (2022.01); A61B 2034/2065 (2016.02); A61B 2034/2068 (2016.02); A61B 2090/365 (2016.02)] 9 Claims
OG exemplary drawing
 
1. A system for generating and providing artificial intelligence assisted surgical guidance, the system comprising:
at least one processor;
a neural network implemented by the at least one processor and trained through analysis of video images from surgical procedures to identify at least one of anatomical objects, surgical objects, and tissue manipulations in the video images;
the neural network being configured to receive, a live feed of video images from a surgery and classify at least one of anatomical objects, surgical objects, and tissue manipulations in the live feed of video images, wherein the neural network includes a backbone comprising a mask-recurrent convolutional neural network (mask RCNN) that receives, as input, a target frame from the live feed of video images and generates as output, image data including a colorization of the target frame that defines likely object contours in terms of color data, the neural network further includes a region proposal network (RPN) that receives, as input, the image data output from the backbone and generates, as output, proposed regions of interest for informing object segmentation, and the neural network further includes mask heads that receive the output from the backbone and the RPN and generates, as outputs, labels for the at least one of the anatomical objects, the surgical objects, and the tissue manipulations in the target frame; and
a surgical guidance generator implemented by the at least one processor for outputting, in real time, guidance based on the classified at least one of anatomical objects, surgical objects, and tissue manipulations in the live feed of video images, wherein outputting the guidance includes overlaying the labels for the at least one of the anatomical objects, the surgical objects, and the tissue manipulations on the live feed of video images or onto a surgical field using augmented reality.