US 12,136,204 B1
Method and apparatus for integrating image enhancement and training data generation using a generative recurrent neural network model
Joon Ki Paik, Seoul (KR)
Assigned to CHUNG ANG UNIVERSITY INDUSTRY ACADEMIC COOPERATION FOUNDATION, Seoul (KR)
Filed by CHUNG ANG UNIVERSITY INDUSTRY ACADEMIC COOPERATION FOUNDATION, Seoul (KR)
Filed on May 13, 2024, as Appl. No. 18/661,749.
Application 18/661,749 is a continuation of application No. PCT/KR2023/021687, filed on Dec. 27, 2023.
Claims priority of application No. 10-2023-0158108 (KR), filed on Nov. 15, 2023.
Int. Cl. G06V 10/771 (2022.01); G06T 7/00 (2017.01)
CPC G06T 7/0002 (2013.01) [G06V 10/771 (2022.01); G06T 2207/20084 (2013.01); G06T 2207/30168 (2013.01)] 8 Claims
OG exemplary drawing
 
1. A method of integrating image enhancement and training data generation using a generative recurrent neural network model, comprising:
(a) receiving a target image as input; and
(b) applying the target image to a trained generative recurrent neural network model to selectively generate one of a high-quality image with enhanced image quality and a low-quality image depending on a type of the target image,
wherein the generative recurrent neural network model comprises:
an image enhancement neural network module that generates the high-quality image with enhanced image quality after receiving the low-quality image among the first image and the second image;
a deteriorated image neural network module that is located behind the image enhancement neural network module, and receives the high-quality image with enhanced image quality and then generates the low-quality image;
a first discrimination module that adjusts a weight of the image enhancement neural network module to minimize first consistency loss for a high-quality image among the first image and the second image and the high-quality image with enhanced image quality; and
a second discrimination module that calculates a second consistency loss using the low-quality image among the first image and the second image and the low-quality image output from the deteriorated image neural network module and then adjusts a weight of the deteriorated image neural network module so that the second consistency loss is minimized,
wherein the image enhancement neural network module and the deteriorated image neural network module are composed of a plurality of transformer-based encoders and a decoder using a plurality of convolutional layers, respectively, and
the plurality of transformers and the plurality of convolution layers are configured to have a symmetrical structure to each other,
wherein each of the transformers comprises:
a convolution layer that extracts a feature map of the input image;
a split layer that splits the extracted feature map;
a first swin transformer block that receives the split feature map and extracts a first global feature map;
a first residual block that is located behind the first swin transformer block and receiving the first global feature map to extract a first local feature map;
a second residual block that receives the split feature map and extracts a second local feature map for the split feature map;
a second swin transformer block that is located behind the second residual block and receives the second local feature map and extracts a second global feature map; and
the first local feature map output from the first residual block and the second global feature map output from the second swin transformer block are combined, and then pass through a convolution layer to output the feature map.