US 11,935,213 B2
Laparoscopic image smoke removal method based on generative adversarial network
Dengwang Li, Shandong (CN); Pu Huang, Shandong (CN); Tingxuan Hong, Shandong (CN); Jie Xue, Shandong (CN); Hua Lu, Shandong (CN); Xueyao Liu, Shandong (CN); Baolong Tian, Shandong (CN); Changming Gu, Shandong (CN); Bin Jin, Shandong (CN); and Xiangyu Zhai, Shandong (CN)
Assigned to Shandong Normal University, Shandong (CN)
Filed by Shandong Normal University, Shandong (CN)
Filed on Mar. 14, 2023, as Appl. No. 18/183,173.
Claims priority of application No. 202210536151.3 (CN), filed on May 18, 2022.
Prior Publication US 2023/0377097 A1, Nov. 23, 2023
Int. Cl. G06T 5/00 (2006.01); G06T 7/00 (2017.01)
CPC G06T 5/002 (2013.01) [G06T 7/0012 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/20172 (2013.01); G06T 2207/30092 (2013.01)] 8 Claims
OG exemplary drawing
 
1. A laparoscopic image smoke removal method based on a generative adversarial network, comprising:
processing a laparoscopic image sample to be processed using a smoke mask segmentation network to acquire a smoke mask image, wherein the smoke mask image comprises smoke position information and smoke density information;
inputting the laparoscopic image sample to be processed and the smoke mask image into a smoke removal network, and extracting features of the laparoscopic image sample to be processed using a multi-level smoke feature extractor to acquire a light smoke feature vector and a heavy smoke feature vector;
wherein a method for extracting the features of the laparoscopic image sample to be processed using the multi-level smoke feature extractor to acquire the light smoke feature vector and the heavy smoke feature vector comprises: the multi-level smoke feature extractor comprising a F1 backbone network, a F2 branch network, and a F3 branch network; inputting the laparoscopic image sample to be processed into the F1 backbone network for a feature extraction; performing two-fold downsampling and four-fold downsampling respectively on the laparoscopic image sample to be processed, and taking sampled images respectively as inputs of the F2 branch network and the F3 branch network; extracting a low-level smoke texture information feature using the F2 branch network; extracting a high-level smoke semantic information feature using the F3 branch network; and acquiring, by the F1 backbone network, the light smoke feature vector and the heavy smoke feature vector according to the low-level smoke texture information feature and the high-level smoke semantic information feature; and
acquiring, according to the light smoke feature vector, the heavy smoke feature vector and the smoke mask image, a smoke-free laparoscopic image by filtering out smoke information and maintaining a laparoscopic image by using a mask shielding effect;
wherein the smoke mask segmentation network and the smoke removal network perform training constraining using a loss function, and the loss function is achieved by the following formula:
LNet=LGAN(Gx,DxS,SF)+λ1Lcyc(Gx,Gy)+λ2Lsap(x)+λ3Ldcp(x)+λ4Lce(x),
where LGAN(Gx, Dx, S, SF) is an adversarial loss used for training the generative adversarial network; Lcyc(Gx, Gy) is a cycle consistency loss used for training a generator network; Ldcp is a dark channel priori loss used for training a discriminator network; Lsap is a smoke area perception loss used for training the smoke mask segmentation network; Lce is a contrast loss; sf is a target domain; s is a source domain; Gx and Gy are respectively a target domain generator and a source domain generator; Dx is a target domain discriminator; and λ1, λ2, λ3, and λ4 are loss function hyperparameters.