US 12,445,656 B2
Guided restoration of video data using neural networks
Debargha Mukherjee, Cupertino, CA (US); Urvang Joshi, Mountain View, CA (US); Yue Chen, Kirkland, WA (US); and Sarah Parker, San Francisco, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by GOOGLE LLC, Mountain View, CA (US)
Filed on Mar. 18, 2022, as Appl. No. 17/698,116.
Application 17/698,116 is a continuation of application No. 16/515,226, filed on Jul. 18, 2019, granted, now 11,282,172, issued on Mar. 22, 2022.
Claims priority of provisional application 62/778,266, filed on Dec. 11, 2018.
Prior Publication US 2022/0207654 A1, Jun. 30, 2022
Int. Cl. H04N 19/82 (2014.01); G06F 18/214 (2023.01); G06N 3/045 (2023.01); G06N 3/08 (2023.01); G06N 20/20 (2019.01); G06T 3/40 (2006.01); G06T 5/50 (2006.01); G06T 5/60 (2024.01); G06T 9/00 (2006.01); H04N 19/117 (2014.01); H04N 19/17 (2014.01); H04N 19/176 (2014.01); H04N 19/70 (2014.01)
CPC H04N 19/82 (2014.11) [G06F 18/214 (2023.01); G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06N 20/20 (2019.01); G06T 3/40 (2013.01); G06T 5/50 (2013.01); G06T 5/60 (2024.01); G06T 9/002 (2013.01); H04N 19/176 (2014.11); H04N 19/70 (2014.11); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); H04N 19/117 (2014.11); H04N 19/17 (2014.11)] 20 Claims
OG exemplary drawing
 
1. A method, comprising:
producing, during a frame decoding process, a reconstructed video frame including degraded video data by dequantizing, inverse transforming, predicting, and loop filtering encoded video data associated with an encoded video frame;
performing a guided restoration process after the loop filtering and during the frame decoding process, wherein performing the guided restoration process includes:
determining a first restoration scheme for restoring a first portion of the degraded video data based on information associated with the first portion, wherein the first restoration scheme indicates to use a first neural network trained on a first value range corresponding to one of a first quantization parameter range or a first non-zero quantized transform coefficient range;
determining a second restoration scheme for restoring a second portion of the degraded video data based on information associated with the second portion, wherein the second restoration scheme indicates to use a second neural network trained on a second value range corresponding to one of a second quantization parameter range or a second non-zero quantized transform coefficient range; and
producing a restored video frame by processing the first portion using the first neural network and by processing the second portion processing the second portion using the second neural network; and
outputting the restored video frame for storage or display.
 
14. A method, comprising:
producing a reconstructed video frame by dequantizing, inverse transforming, predicting, and loop filtering encoded video data associated with an encoded video frame;
performing, after the loop filtering, a guided restoration process including:
determining restoration schemes for restoring degraded video data of separate portions of the reconstructed video frame based on information associated with the separate portions, wherein each restoration scheme indicates to use a different neural network trained on a different value range each corresponding to one of a quantization range or a non-zero quantized transform coefficient range; and
producing a restored video frame by processing each portion of the separate portions using a neural network associated with a respective one of the restoration schemes determined for the portion; and
outputting the restored video frame for storage or display.