US 12,223,621 B2
Virtual viewpoint synthesis method, electronic apparatus, and computer readable medium
Yan Sun, Beijing (CN); Minglei Chu, Beijing (CN); Tiankuo Shi, Beijing (CN); Yanhui Xi, Beijing (CN); Zhihua Ji, Beijing (CN); Yifan Hou, Beijing (CN); Chenxi Zhao, Beijing (CN); Shuo Zhang, Beijing (CN); Xiangjun Peng, Beijing (CN); Xiaomang Zhang, Beijing (CN); and Wei Sun, Beijing (CN)
Assigned to BEIJING BOE OPTOELECTRONICS TECHNOLOGY CO., LTD., Beijing (CN); and BOE TECHNOLOGY GROUP CO., LTD., Beijing (CN)
Appl. No. 17/614,120
Filed by Beijing BOE Optoelectronics Technology Co., Ltd., Beijing (CN); and BOE Technology Group Co., Ltd., Beijing (CN)
PCT Filed Jan. 25, 2021, PCT No. PCT/CN2021/073565
§ 371(c)(1), (2) Date Nov. 24, 2021,
PCT Pub. No. WO2022/155950, PCT Pub. Date Jul. 28, 2022.
Prior Publication US 2023/0162338 A1, May 25, 2023
Int. Cl. G06T 5/50 (2006.01); G06T 7/13 (2017.01); G06T 7/55 (2017.01)
CPC G06T 5/50 (2013.01) [G06T 7/13 (2017.01); G06T 7/55 (2017.01); G06T 2207/10028 (2013.01); G06T 2207/20221 (2013.01)] 15 Claims
OG exemplary drawing
 
1. A virtual viewpoint synthesis method, comprising:
Pre-processing a depth image with zero parallax corresponding to an original image to obtain a processed depth image;
generating virtual viewpoint images corresponding to a plurality of virtual viewpoints respectively according to the processed depth image and the original image; and
filling holes in the virtual viewpoint images to generate a plurality of filled virtual viewpoint images,
wherein the pre-processing the depth image with zero parallax corresponding to the original image to obtain the processed depth image comprises:
determining edge pixels in the depth image with zero parallax according to gradients of pixels in the depth image with zero parallax in a row direction;
determining n pixels adjacent to the edge pixels in the row direction as target pixels; and
adjusting a depth value of each target pixel according to a maximum depth value of preset neighborhoods corresponding to the target pixels,
the filling the holes in the virtual viewpoint images and generating the plurality of filled virtual viewpoint images comprises:
traversing all pixels of a background area in a target virtual viewpoint image to determine a hole of the background area, wherein the target virtual viewpoint image is any one of the virtual viewpoint images, and depth values corresponding to all pixels of the background area are less than or equal to a preset depth value threshold; and
filling, for any hole in the background area, the hole according to pixel information of at least part of pixels adjacent to the hole in the row direction,
the filling, for any hole in the background area, the hole based on the pixel information of at least part of pixels adjacent to the hole in the row direction comprises:
filling, for a row pixel corresponding to any row in the hole, the row pixel according to pixel information of two pixels which are closest to the row pixel in the row direction, and
wherein the filling, for the row pixel corresponding to any row in the hole, the row pixel according to pixel information of two pixels which are closest to the row pixel in the row direction comprises:
in response to that the two pixels belong to a same parallax plane, filling the row pixel by a linear interpolation according to the pixel information of the two pixels; and
in response to that the two pixels do not belong to the same parallax plane, filling the row pixel according to the pixel information of one of the two pixels with a minimum depth value.