| CPC A61B 1/000095 (2022.02) [A61B 1/000096 (2022.02); A61B 1/3132 (2013.01); G06T 5/20 (2013.01); G06T 5/50 (2013.01); G06T 5/60 (2024.01); G06T 5/70 (2024.01); G06T 5/92 (2024.01); G06T 7/0014 (2013.01); G06T 7/11 (2017.01); G06T 7/174 (2017.01); G06T 11/00 (2013.01); G06T 2207/10016 (2013.01); G06T 2207/20076 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/20182 (2013.01); G06T 2207/30004 (2013.01); G06T 2210/41 (2013.01)] | 6 Claims |
|
1. A method for removing smoke from laparoscope images based on a conditional diffusion model, comprising the following steps:
1) segmenting a video of a laparoscopic surgery according to the number of frames to form a data set in the form of pictures; performing smoke rendering on the obtained laparoscope smokeless images, and synthesizing paired smoky images to obtain a synthetic data set containing the smokeless images and the smoky images;
2) inputting the smokeless images into the conditional diffusion model for forward noise addition, and continuously adding noise until the smokeless images are completely noised to obtain a series of noisy images,
wherein a specific operation of the step 2) is as follows: inputting the smokeless images captured by a laparoscope into the conditional diffusion model for forward noise addition, wherein an added noise variance is βt; the noise addition process follows a Markov chain, and a noise-added image is recorded as xt; and performing noise addition on inputted smokeless images T times until the smokeless images are completely noised to obtain the series of noisy images {x1, x2, . . . , xt, . . . , xT}, wherein xt represents a noise image obtained by performing a t-th noise addition, and xT represents a noise image obtained by performing a T-th noise addition;
an equation for the forward noise addition process is as follows:
 wherein ε represents noise, N represents a standard normal Gaussian distribution, I represents an identity matrix, and xt-1 represents a noise image obtained by performing a t−1-th noise addition;
3) inputting the smoky images into a smoke sensing module to obtain smoke concentration and position information, then training a neural network through the series of noisy images, and performing reverse denoising using the trained neural network; continuously performing reverse denoising on the completely noised images obtained in the step 2) until clear smokeless images are outputted,
wherein a specific operation of the step 3) is as follows: inputting the smoky images in the synthetic data set into the smoke sensing module to obtain smoke mask information and dark channel prior (DCP) information; taking the series of noisy images in the step 2) as labels for reverse training, wherein the reverse training process of each label is carried out through a U-Net network with the addition of a feature frequency compensation module FCB to jointly complete the training of the U-Net network; taking guidance images and the series of noisy images as inputs of the reverse training to train the U-Net network, the guidance images comprising the smokeless images and the synthetic smoky images in the synthetic data set; and finally, performing reverse denoising using the trained neural network, taking the smoke mask information and the DCP information as condition information to guide the reverse denoising process, and continuously performing denoising on the inputted completely noised images finally obtained in the step 2) until clear laparoscope smokeless images are obtained;
the smoke sensing module comprises a smoke mask segmentation module and a DCP module, for smoke segmentation and smoke concentration information extraction on the inputted smoky images, respectively;
minimum values in three channels of R, G, and B are taken to form a grayscale image, and then minimum value filtering is performed to obtain a dark channel in the DCP module; then, a network acquires smoke distribution and concentration information in the smoke images through the DCP, and the obtained smoke information is used as condition information to guide the reverse denoising in the diffusion model to generate related smokeless images;
related calculation equations are as follows:
Mc(F)=σ(MLP(AvgPool(F)+MLP(MaxPool(F))),
Ms(F)=σ(f7×7([AvgPool(F);MaxPool(F)])),
F′=Mc⊙(Mc⊙F),
wherein Mc(F) represents channel attention, Ms(F) represents spatial attention, F represents a depth feature, σ represents a sigmoid activation function, AvgPool represents average pooling, MaxPool represents maximum pooling, MLP represents an activation function, f7×7 represents a convolution operation of a 7+7 filter, [⋅, ⋅] represents connection of feature maps, (represents element multiplication, and F′ represents a feature map;
the feature frequency compensation module FCB comprises a plurality of convolution filters, and bandwidth of these filters covers mid- and high-frequency components that are difficult to capture in the network; the calculation of the feature frequency compensation module FCB is as follows:
fk,σ=Gk×kσ*f, k∈{3,5,7,9 . . . },
wherein Gk×kσ represents a two-dimensional Gaussian kernel having a mean value of σ and a size of k, f represents an inputted feature value, and fk,σ represents a convolution output through the two-dimensional Gaussian kernel; four Gaussian kernels having sizes of 3, 5, 7, and 9 are selected for filtering, and a filter is obtained after making a difference between two Gaussian kernels, which is calculated and expressed by an equation as:
fk′={fk′,f3′,f5′,f7′,f9′}={f,f−f3,f3−f5,f5−f7,f7−f9},
wherein fk′ represents an output after filtering, f9′ represents frequency information after being filtered by the Gaussian kernel having a size of 9, f7′ represents frequency information after being filtered by the Gaussian kernel having a size of 7, f5′ represents frequency information after being filtered by the Gaussian kernel having a size of 5, and f3′ represents frequency information after being filtered by the Gaussian kernel having a size of 3;
filtering of different frequency bands is then realized by selecting different Gaussian kernels, and f is weighted and summarized as follows:
 wherein W=[W1, W2, W3, W4, W5] represents a weight of trainable learning, and f represents an output after being weighted and summarized; and
4) optimizing a smoke removal model through a multi-loss function fusion strategy, and accelerating the model to generate smoke removal images using a skip sampling strategy.
|