US 12,003,885 B2
Video frame interpolation via feature pyramid flows
Luming Liang, Redmond, WA (US); Tianyu Ding, Baltimore, MD (US); and Ilya Dmitriyevich Zharkov, Sammamish, WA (US)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on Jun. 14, 2021, as Appl. No. 17/347,481.
Prior Publication US 2022/0400226 A1, Dec. 15, 2022
Int. Cl. G06T 3/4007 (2024.01); G06N 3/045 (2023.01); G06V 10/46 (2022.01); H04N 7/01 (2006.01); H04N 19/587 (2014.01)
CPC H04N 7/014 (2013.01) [G06N 3/045 (2023.01); G06T 3/4007 (2013.01); G06V 10/462 (2022.01); H04N 7/0127 (2013.01); H04N 7/0145 (2013.01); H04N 19/587 (2014.11)] 18 Claims
OG exemplary drawing
 
1. A convolutional neural network configured to generate an interpolated image based on a first image and a second image, the convolutional neural network including:
a first network including an encoder and a decoder, the first network configured to:
extract first features from the first image, and
extract second features from the second image;
a feature pyramid configured to:
extract multi-scale features from encoded first features, and
extract multi-scale features from encoded second features; and
a synthesis network configured to:
warp the first features from the first image using a first plurality of parameters,
warp the multi-scale features extracted from the encoded first features using the first plurality of parameters,
warp the second features from the second image using a second plurality of parameters,
warp the multi-scale features extracted from the encoded second features using the second plurality of parameters,
generate a first candidate intermediate frame based on the warped first features and the warped second features,
generate a second candidate intermediate frame based on the warped multi-scale features extracted from the encoded first features and the warped multi-scale features extracted from the encoded second features, and
blend the first candidate intermediate frame and the second candidate intermediate frame to generate an interpolated frame.