US 12,329,491 B2
Method and system for augmented imaging in open treatment using multispectral information
Lena Maier-Hein, Heidelberg (DE); Sebastian Josef Wirkert, Heidelberg (DE); Anant Suraj Vemuri, Heidelberg (DE); Leonardo Antonio Ayala Menjivar, Heidelberg (DE); Silvia Seidlitz, Heidelberg (DE); Thomas Kirchner, Heidelberg (DE); and Tim Adler, Heidelberg (DE)
Assigned to DEUTSCHES KREBSFORSCHUNGSZENTRUM STIFTUNG DES ÖFFENTLICHEN RECHTS, Heidelberg (DE)
Appl. No. 17/263,850
Filed by DEUTSCHES KREBSFORSCHUNGSZENTRUM STIFTUNG DES ÖFFENTLICHEN RECHTS, Heidelberg (DE)
PCT Filed Jul. 31, 2019, PCT No. PCT/EP2019/070649
§ 371(c)(1), (2) Date Jan. 27, 2021,
PCT Pub. No. WO2020/025684, PCT Pub. Date Feb. 6, 2020.
Claims priority of application No. 18186670 (EP), filed on Jul. 31, 2018.
Prior Publication US 2022/0008157 A1, Jan. 13, 2022
Int. Cl. A61B 5/00 (2006.01); A61B 5/02 (2006.01); A61B 5/0205 (2006.01); A61B 5/145 (2006.01); A61B 5/1455 (2006.01); A61B 90/00 (2016.01); G06F 18/2413 (2023.01); G06T 7/00 (2017.01); G06V 10/141 (2022.01); G06V 10/46 (2022.01); G06V 10/60 (2022.01); G06V 10/764 (2022.01); G16H 20/40 (2018.01); G16H 30/40 (2018.01); G16H 50/20 (2018.01); G16H 50/70 (2018.01)
CPC A61B 5/0037 (2013.01) [A61B 5/0002 (2013.01); A61B 5/02042 (2013.01); A61B 5/0205 (2013.01); A61B 5/14546 (2013.01); A61B 5/14551 (2013.01); A61B 5/443 (2013.01); A61B 5/4547 (2013.01); A61B 5/4875 (2013.01); A61B 5/489 (2013.01); A61B 5/4893 (2013.01); A61B 5/7275 (2013.01); A61B 5/749 (2013.01); A61B 90/361 (2016.02); G06F 18/2413 (2023.01); G06T 7/0012 (2013.01); G06V 10/141 (2022.01); G06V 10/60 (2022.01); G06V 10/764 (2022.01); G16H 20/40 (2018.01); G16H 30/40 (2018.01); G16H 50/20 (2018.01); G16H 50/70 (2018.01); A61B 2090/365 (2016.02); G06T 2207/20081 (2013.01); G06V 10/467 (2022.01); G06V 2201/031 (2022.01)] 24 Claims
OG exemplary drawing
 
1. A method of generating and displaying one or more augmented images of tissue of a patient undergoing open treatment, said method comprising:
estimating a spectral composition of light illuminating a region of interest of the tissue by measurement or by computing said spectral composition based on control parameters controlling said light illuminating the region of interest,
obtaining one or more multispectral images of the region of interest using a multispectral camera,
applying said one or more multispectral images or one or more images derived from said one or more multispectral images to a machine learning based regressor or classifier to thereby derive one or more tissue parameters associated with image regions or pixels of the one or more multispectral images, wherein said machine learning based regressor or classifier has been trained to predict the one or more tissue parameters from the one or more multispectral images under a given spectral composition of illumination, and
generating one or more augmented images and displaying said one or more augmented images on a display device, wherein said one or more augmented images correspond to the one or more multispectral images and include the one or more tissue parameters,
wherein the machine learning based regressor or classifier is made to match the estimated spectral composition of light illuminating said region of interest of the tissue by one of
selecting the machine learning based regressor or classifier from among a plurality of candidate machine learning based regressors or classifiers that have each been previously trained for a different spectral composition of illumination, such that the given spectral composition of illumination for the machine learning based regressor or classifier is the most similar to the estimated spectral composition of said light illuminating said region of interest, or
transforming the obtained one or more multispectral images based on information derived from the estimated spectral composition of the light illuminating said region of interest and applying the transformed one or more multispectral images to a standard machine learning based regressor or classifier, which has been trained under a standard illumination, wherein the transformation is for compensating for a change in the one or more multispectral images due to a deviation in the spectral composition of the illumination from the standard illumination, or
retraining an already trained machine learning based regressor or classifier using simulation data that is adapted to the estimated spectral composition of light illuminating said region of interest.