US 12,464,139 B2
Video inter/intra compression using mixture of experts
David Charles Minnen, Mountain View, CA (US); Sung Jin Hwang, Mountain View, CA (US); Elliott Michael Karpilovsky, Santa Clara, CA (US); and Debargha Mukherjee, Cupertino, CA (US)
Assigned to Google LLC, Mountain View, CA (US)
Appl. No. 18/286,574
Filed by Google LLC, Mountain View, CA (US)
PCT Filed May 7, 2021, PCT No. PCT/US2021/031218
§ 371(c)(1), (2) Date Oct. 12, 2023,
PCT Pub. No. WO2022/235270, PCT Pub. Date Nov. 10, 2022.
Prior Publication US 2024/0195985 A1, Jun. 13, 2024
Int. Cl. H04N 19/176 (2014.01); H04N 19/119 (2014.01); H04N 19/159 (2014.01)
CPC H04N 19/176 (2014.11) [H04N 19/119 (2014.11); H04N 19/159 (2014.11)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method performed by a data processing apparatus, comprising:
for each frame in a video that includes multiple frames in an order sequence, defining, in the frame, a plurality of blocks in the frame, where each block in the frame is a sub-portion of the frame;
processing the frames of video in sequential sets, wherein each sequential set is at least a current frame of video and a prior frame of video in the ordered sequence;
for each sequential set, predicting blocks in the current frame of the video, each respective prediction of a block in the frame of the video comprising providing, as input to a prediction model:
a first border of a current block of the current frame of the video;
a second border of a current block of the current frame of the video;
a first border for a respective current block of the prior frame of the video;
a second border for the respective current block of the prior frame of the video;
the respective current block of the prior frame of the video; and
predicting, by the prediction model and based on the input to the prediction model for the sequential set, at least a sub-portion of the current block of the current frame of video;
generating multiple compressed representations of a same block of the current frame, wherein each given compressed representation among the multiple compressed representations is generated by a different encoder expert model among multiple encoder expert models; and
selecting, by a gating apparatus, one multiple compressed representation from among the multiple compressed representations as a final compressed representation of the same block.