US 12,482,060 B2
Panoramic video frame interpolation method and apparatus, and corresponding storage medium
Dan Chen, Shenzhen (CN); Yuyao Zhang, Shenzhen (CN); and Zhigang Tan, Shenzhen (CN)
Assigned to KANDAO TECHNOLOGY CO., LTD., Shenzhen (CN)
Appl. No. 18/041,802
Filed by KANDAO TECHNOLOGY CO., LTD., Shenzhen (CN)
PCT Filed Apr. 19, 2021, PCT No. PCT/CN2021/088004
§ 371(c)(1), (2) Date Feb. 16, 2023,
PCT Pub. No. WO2021/238500, PCT Pub. Date Dec. 2, 2021.
Claims priority of application No. 202010452273.5 (CN), filed on May 26, 2020.
Prior Publication US 2023/0316456 A1, Oct. 5, 2023
Int. Cl. G06T 3/4038 (2024.01); G06T 3/4007 (2024.01); G06T 3/4046 (2024.01); G06T 5/10 (2006.01); G06T 5/50 (2006.01); G06T 7/20 (2017.01)
CPC G06T 3/4038 (2013.01) [G06T 3/4007 (2013.01); G06T 3/4046 (2013.01); G06T 5/10 (2013.01); G06T 5/50 (2013.01); G06T 7/20 (2013.01); G06T 2207/10016 (2013.01); G06T 2207/20048 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/20212 (2013.01)] 11 Claims
OG exemplary drawing
 
1. A panoramic video frame interpolation method, comprising:
performing downsampling processing on a panoramic video pre-frame image to obtain a downsampling pre-frame image; performing the downsampling processing on a panoramic video post-frame image to obtain a downsampling post-frame image;
inputting the downsampling pre-frame image and the downsampling post-frame image to a preset optical flow estimation neural network to output a pre-post frame image optical flow graph and a post-pre frame image optical flow graph;
calculating an image optical flow graph before frame interpolation and an image optical flow graph after frame interpolation based on a frame interpolation position, the pre-post frame image optical flow graph and the post-pre frame image optical flow graph;
performing an inverse transformation operation on the downsampling post-frame image by using the pre-post frame image optical flow graph to obtain a downsampling pre-frame image after transformation; performing the inverse transformation operation on the downsampling pre-frame image by using the post-pre frame image optical flow graph to obtain a downsampling post-frame image after transformation;
inputting the downsampling pre-frame image, the downsampling post-frame image, the pre-post frame image optical flow graph, the post-pre frame image optical flow graph, the image optical flow graph before frame interpolation, the image optical flow graph after frame interpolation, the downsampling pre-frame image after transformation and the downsampling post-frame image after transformation to a preset optical flow correction neural network to output a pre-post frame image correction optical flow graph, a post-pre frame image correction optical flow graph and an image occlusion relationship graph;
correcting the downsampling pre-frame image by using the pre-post frame image correction optical flow graph to obtain a downsampling pre-frame image after correction; correcting the downsampling post-frame image by using the post-pre frame image correction optical flow graph to obtain a downsampling post-frame image after correction;
performing upsampling processing on the image optical flow graph before frame interpolation to obtain an upsampling image optical flow graph before frame interpolation; performing the upsampling processing on the image optical flow graph after frame interpolation to obtain an upsampling image optical flow graph after frame interpolation; and
calculating a frame interpolation image corresponding to the frame interpolation position by using the downsampling pre-frame image after correction, the downsampling post-frame image after correction, the image occlusion relationship graph, the upsampling image optical flow graph before frame interpolation and the upsampling image optical flow graph after frame interpolation.