US 11,842,460 B1
Burst image fusion and denoising using end-to-end deep neural networks
Chen Chen, Santa Clara, CA (US); Gustav M. Larsson, San Francisco, CA (US); Jiangkun Liu, San Jose, CA (US); Srikrishna Sridhar, Seattle, WA (US); and Abdelrahman Abdelhamed, Toronto (CA)
Assigned to Apple Inc., Cupertino, CA (US)
Filed by Apple Inc., Cupertino, CA (US)
Filed on Jun. 18, 2021, as Appl. No. 17/351,820.
Claims priority of provisional application 63/041,445, filed on Jun. 19, 2020.
Int. Cl. G06T 5/00 (2006.01); G06T 5/50 (2006.01)
CPC G06T 5/002 (2013.01) [G06T 5/50 (2013.01); G06T 2200/24 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/20182 (2013.01); G06T 2207/20221 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A device, comprising:
a memory;
one or more image capture devices;
a user interface; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain a first neural network having a first network architecture, wherein the first network architecture is configured to perform a fusion operation and a denoising operation on sets of input images, and wherein the first network architecture comprises:
a first plurality of network layers configured to compute optical flow information between a first set of input images;
a second plurality of network layers configured to perform, at least in part, the fusion and denoising operations on the first set of input images; and
a third plurality of skip connections between layers of the second plurality of network layers, wherein parameters for each skip connection of the third plurality of skip connections are warped based on at least part of the optical flow information computed by the first plurality of network layers; and
use the first neural network to perform the fusion and denoising operations on the first set of input images.