US 11,657,087 B2
Surgical video retrieval based on preoperative images
Martin Habbecke, Palo Alto, CA (US); Joëlle K. Barral, Mountain View, CA (US); Lin Yang, Sunnyvale, CA (US); and Xing Jin, San Jose, CA (US)
Assigned to Verily Life Sciences LLC, South San Francisco, CA (US)
Filed by Verily Life Sciences LLC, South San Francisco, CA (US)
Filed on Mar. 12, 2019, as Appl. No. 16/299,965.
Claims priority of provisional application 62/644,737, filed on Mar. 19, 2018.
Prior Publication US 2019/0286652 A1, Sep. 19, 2019
Int. Cl. G06F 16/783 (2019.01); G06F 16/71 (2019.01); A61B 90/00 (2016.01); G06N 20/00 (2019.01); G16H 10/60 (2018.01); A61B 34/20 (2016.01); G16H 30/20 (2018.01); G16H 20/40 (2018.01); A61B 34/10 (2016.01); G09B 23/28 (2006.01); G06V 10/70 (2022.01); G06N 3/08 (2023.01); G06T 7/00 (2017.01); G16H 30/40 (2018.01); G16H 50/70 (2018.01)
CPC G06F 16/7837 (2019.01) [A61B 34/10 (2016.02); A61B 34/20 (2016.02); A61B 90/37 (2016.02); G06F 16/71 (2019.01); G06F 16/7847 (2019.01); G06N 3/08 (2013.01); G06N 20/00 (2019.01); G06V 10/70 (2022.01); G09B 23/28 (2013.01); G16H 10/60 (2018.01); G16H 20/40 (2018.01); G16H 30/20 (2018.01); G06T 7/0012 (2013.01); G06T 2207/10016 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/30004 (2013.01); G16H 30/40 (2018.01); G16H 50/70 (2018.01)] 23 Claims
OG exemplary drawing
 
12. A method for the retrieval of surgical videos, comprising:
receiving, with a processor coupled to memory and a database, one or more input preoperative images of organs and disease in a human body, wherein the one or more input preoperative images of the organs and the disease were captured with a medical imaging device, and wherein the database includes surgical videos;
determining, with the processor, first organ information and first disease information from the one or more input preoperative images using a machine learning algorithm included in the memory to generate a feature vector associated with the one or more input preoperative images, wherein the first organ information includes at least one of a shape of the organs, a size of the organs, or a location of the organs, and wherein the first disease information includes at least one of a shape of the disease, a location of the disease, a size of the disease, or a stage of the disease, and wherein the feature vector includes the first organ information and the first disease information;
calculating, with the processor, a similarity score between the one or more input preoperative images and the surgical videos included in the database, wherein the similarity score corresponds to a comparison between the feature vector associated with the one or more input preoperative images and corresponding feature vectors associated with second disease information and second organ information indexed to the surgical videos;
selecting, with the processor, one or more of the surgical videos from the database based on the similarity score for display;
inputting the one or more input preoperative images into the machine learning algorithm to determine a position of the one or more input preoperative images within a feature space, wherein the feature vector includes the position; and
determining a distance from the position of the one or more input preoperative images within the feature space to corresponding positions within the feature space for surgical videos included in the database when calculating the similarity score, wherein the similarity score is based, at least in part, on the distance.