US 11,689,726 B2
Hybrid motion-compensated neural network with side-information based video coding
Debargha Mukherjee, Cupertino, CA (US); Urvang Joshi, Mountain View, CA (US); Yue Chen, Kirkland, WA (US); and Sarah Parker, San Francisco, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by GOOGLE LLC, Mountain View, CA (US)
Filed on Jul. 19, 2019, as Appl. No. 16/516,784.
Claims priority of provisional application 62/775,481, filed on Dec. 5, 2018.
Prior Publication US 2020/0186809 A1, Jun. 11, 2020
Int. Cl. H04N 19/147 (2014.01); H04N 19/184 (2014.01); H04N 19/59 (2014.01); G06N 3/04 (2023.01)
CPC H04N 19/147 (2014.11) [G06N 3/04 (2013.01); H04N 19/184 (2014.11); H04N 19/59 (2014.11)] 20 Claims
OG exemplary drawing
 
1. A hybrid apparatus for coding a video stream, comprising:
a first encoder that receives source data from the video stream and receives side information correlated with the source date, the first encoder comprising:
a neural network having at least one hidden layer, wherein the neural network:
receives the source data at a first hidden layer of the at least one hidden layer;
receives the side information at the first hidden layer; and
generates guided information using the source data and the side information; and
an entropy coder that entropy encodes block data from the source data into a compressed bitstream, wherein:
the first encoder outputs the compressed bitstream from the entropy coder to a decoder,
the neural network outputs the guided information, and
the first encoder, in addition to providing the side information to the neural network, outputs the side information separately from the compressed bitstream so that the side information bypasses the neural network for the decoder to reconstruct the source data from the compressed bitstream and the side information to produce reconstructed source data.