US 12,095,983 B2
Techniques for deep neural network based inter- frame prediction in video coding
Zeqiang Li, Palo Alto, CA (US); Byeongdoo Choi, Palo Alto, CA (US); Wei Wang, Palo Alto, CA (US); Wei Jiang, Sunnyvale, CA (US); Xiaozhong Xu, State College, PA (US); and Shan Liu, San Jose, CA (US)
Assigned to TENCENT AMERICA LLC, Palo Alto, CA (US)
Filed by TENCENT AMERICA LLC, Palo Alto, CA (US)
Filed on Sep. 8, 2022, as Appl. No. 17/940,690.
Application 17/940,690 is a continuation of application No. 17/476,928, filed on Sep. 16, 2021, granted, now 11,490,078.
Claims priority of provisional application 63/131,625, filed on Dec. 29, 2020.
Prior Publication US 2023/0007240 A1, Jan. 5, 2023
Int. Cl. H04N 19/105 (2014.01); G06N 3/045 (2023.01); H04N 19/172 (2014.01); H04N 19/182 (2014.01); H04N 19/50 (2014.01)
CPC H04N 19/105 (2014.11) [G06N 3/045 (2023.01); H04N 19/172 (2014.11); H04N 19/182 (2014.11); H04N 19/50 (2014.11)] 20 Claims
OG exemplary drawing
 
1. A method of video coding using neural network based inter-frame prediction, the method being performed by at least one processor, and the method comprising:
generating reference picture lists including a plurality of reference pictures;
determining a presence of first reference pictures included in the reference picture lists based on a picture order count (POC) distance;
selecting one or more different network models, based on the determination of the presence of the first reference pictures;
generating intermediate flows from a perspective of the first reference pictures and iteratively updating flow fields of the intermediate flows based on the first reference pictures;
generating an intermediate reference picture based on the selected one or more different neural network models and the intermediate flows;
predicted a picture based on a weight of a feature map by refine the intermediate reference picture, the feature map including a plurality of levels; and
computing an enhanced picture as an output by adding a final residual, based on the predicted picture, to the first reference pictures.