US 12,347,216 B2
Interactive endoscopy for intraoperative virtual annotation in vats and minimally invasive surgery
Paul Thienphrapa, Cambridge, MA (US); Torre Michelle Bydlon, Melrose, MA (US); Alvin Chen, Cambridge, MA (US); Prasad Vagdargi, Cambridge, MA (US); and William McNamara, Valhalla, NY (US)
Assigned to KONINKLIJKE PHILIPS N.V., Eindhoven (NL)
Appl. No. 17/641,940
Filed by KONINKLIJKE PHILIPS N.V., Eindhoven (NL)
PCT Filed Sep. 11, 2020, PCT No. PCT/EP2020/075422
§ 371(c)(1), (2) Date Mar. 10, 2022,
PCT Pub. No. WO2021/048326, PCT Pub. Date Mar. 18, 2021.
Claims priority of provisional application 62/899,365, filed on Sep. 12, 2019.
Prior Publication US 2022/0358773 A1, Nov. 10, 2022
Int. Cl. G06V 20/70 (2022.01); A61B 1/00 (2006.01); G06T 7/00 (2017.01); G06T 7/13 (2017.01); G06T 7/246 (2017.01); G06T 7/30 (2017.01); G06T 7/73 (2017.01); G06T 11/00 (2006.01); G06V 10/44 (2022.01); G06V 10/764 (2022.01)
CPC G06V 20/70 (2022.01) [A61B 1/000094 (2022.02); A61B 1/000096 (2022.02); A61B 1/0005 (2013.01); G06T 7/0012 (2013.01); G06T 7/13 (2017.01); G06T 7/246 (2017.01); G06T 7/30 (2017.01); G06T 7/73 (2017.01); G06T 11/00 (2013.01); G06V 10/44 (2022.01); G06V 10/764 (2022.01); G06T 2207/10016 (2013.01); G06T 2207/10068 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/30064 (2013.01); G06T 2207/30096 (2013.01); G06V 2201/031 (2022.01); G06V 2201/034 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A controller for live annotation of interventional imagery, the controller comprising:
a processor in communication with memory, the processor configured to:
receive interventional imagery during an intraoperative intervention;
automatically analyze the interventional imagery for detectable features;
detect a detectable feature in the interventional imagery;
determine to add an annotation to the interventional imagery for the detectable feature;
identify, based on inputting the detectable feature to machine learning, an optimized location to add the annotation to the interventional imagery;
add the annotation to the interventional imagery at the identified optimized location to correspond to the detectable feature;
output, during the intraoperative intervention, a video output based on the interventional imagery and the annotation, the video output including the annotation overlaid on the interventional imagery at the identified optimized location; and
apply the machine learning to the video output to obtain quantitative metrics of at least one of (i) one or more features in the video output and (ii) one or more annotations in the video output, wherein
addition of the annotation at the optimized location in the video output is based on the quantitative metrics from the machine learning from previous applications of the machine learning to previous video output.