US 11,871,010 B2
Extended low-frequency non-separable transform (LFNST) designs with worst-case complexity handling
Hilmi Enes Egilmez, San Diego, CA (US); Vadim Seregin, San Diego, CA (US); and Marta Karczewicz, San Diego, CA (US)
Assigned to QUALCOMM Incorporated, San Diego, CA (US)
Filed by QUALCOMM Incorporated, San Diego, CA (US)
Filed on May 21, 2021, as Appl. No. 17/326,588.
Claims priority of provisional application 63/086,888, filed on Oct. 2, 2020.
Prior Publication US 2022/0109858 A1, Apr. 7, 2022
Int. Cl. H04N 19/18 (2014.01); H04N 19/176 (2014.01); H04N 19/159 (2014.01); H04N 19/117 (2014.01)
CPC H04N 19/18 (2014.11) [H04N 19/117 (2014.11); H04N 19/159 (2014.11); H04N 19/176 (2014.11)] 28 Claims
OG exemplary drawing
 
1. A method of decoding video data, the method comprising:
determining a number of allowed non-zero coefficients for a block of video data based on a size of the block;
obtaining a set of dequantized coefficients for the block of video data, wherein the set of dequantized coefficients comprises a first subset of dequantized coefficients that includes non-zero dequantized coefficients and a second subset of dequantized coefficients that includes all zero coefficients
determining an intra prediction mode for the block of video data;
based on the intra prediction mode, determining a set of inverse LFNST candidates from a plurality of sets, wherein the plurality of sets includes 35 sets, and the determining comprising determining an index indicative of the set of inverse LFNST candidates from the plurality of sets based on the inter prediction mode, and wherein determining the index based on the intra prediction mode comprises:
for values of the intra prediction mode between 0 and 34, determining the index to be equal to the value of the intra prediction mode; and
for values of the intra prediction mode between 35 and 66, determining the index to be equal to 68 minus the value of the intra prediction mode;
selecting an inverse LFNST from the determined set of inverse LFNST candidates,
applying the inverse low-frequency non-separable transform (LFNST) to the first subset of dequantized coefficients to determine a first intermediate subset of coefficients; and
applying an inverse separable transform to the first intermediate subset of coefficients and at least a portion of the second subset of coefficients to determine a block of reconstructed residual values.