US 11,869,170 B2
Generating super-resolution images using neural networks
David Berthelot, Mountain View, CA (US); and Ian Goodfellow, Mountain View, CA (US)
Assigned to Google LLC, Mountain View, CA (US)
Appl. No. 17/293,754
Filed by Google LLC, Mountain View, CA (US)
PCT Filed Nov. 18, 2019, PCT No. PCT/US2019/062041
§ 371(c)(1), (2) Date May 13, 2021,
PCT Pub. No. WO2020/102812, PCT Pub. Date May 22, 2020.
Claims priority of provisional application 62/768,801, filed on Nov. 16, 2018.
Prior Publication US 2021/0407042 A1, Dec. 30, 2021
Int. Cl. G06T 3/40 (2006.01); G06N 3/08 (2023.01); G06T 5/50 (2006.01); G06F 18/22 (2023.01); G06N 3/045 (2023.01)
CPC G06T 3/4046 (2013.01) [G06F 18/22 (2023.01); G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06T 3/4053 (2013.01); G06T 5/50 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of training a super-resolution neural network having a plurality of super-resolution parameters, wherein the super-resolution neural network is configured to receive a network input comprising an input image having a first resolution and to process the network input to generate a super-resolution output image that is a version of the input image with a second, higher resolution, wherein the network input includes a vector representing a desired location in a space of possible super-resolution output images, and the method comprising:
receiving a training input image and a ground truth super-resolution output image for the training image;
processing a first training network input comprising the training input image using the super-resolution neural network and in accordance with current values of the super-resolution parameters to generate a first training super-resolution output image, wherein the first training network input includes a first vector representing a location of the ground truth super-resolution output image in the space of possible super-resolution output images;
processing a first critic input generated from (i) the training input image and (ii) the ground truth super-resolution output image using a critic neural network having a plurality of critic parameters and in accordance with current values of the critic parameters, wherein the critic neural network is configured to receive the first critic input and to process the first critic input in accordance with the current values of the critic parameters to:
map the first critic input to a latent representation of the first critic input in a perceptual latent space;
processing a second critic input generated from (i) the training input image and (ii) the first training super-resolution output image generated by the super-resolution neural network using the critic neural network and in accordance with the current values of the critic parameters to map the second critic input to a latent representation of the second critic input in the perceptual latent space;
determining a gradient with respect to the super-resolution parameters of a generator loss function that includes a perceptual loss that measures a distance between the latent representation of the first critic input and the latent representation of the second critic input; and
determining, from the gradient, an update to the current values of the super-resolution parameters.