US 10,893,250 B2
Free-viewpoint photorealistic view synthesis from casually captured video
Abhishek Kar, Berkeley, CA (US); Rodrigo Ortiz Cayon, San Francisco, CA (US); Ben Mildenhall, Berkeley, CA (US); Stefan Johannes Josef Holzer, San Mateo, CA (US); and Radu Bogdan Rusu, San Francisco, CA (US)
Assigned to Fyusion, Inc., San Francisco, CA (US)
Filed by Fyusion, Inc., San Francisco, CA (US)
Filed on Sep. 18, 2019, as Appl. No. 16/574,652.
Claims priority of provisional application 62/792,163, filed on Jan. 14, 2019.
Prior Publication US 2020/0226816 A1, Jul. 16, 2020
Int. Cl. G09G 5/00 (2006.01); H04N 13/111 (2018.01); G06T 7/557 (2017.01); H04N 13/271 (2018.01); G06T 7/579 (2017.01); G06T 7/70 (2017.01); G06T 7/514 (2017.01); G06T 15/20 (2011.01); G06T 7/00 (2017.01); G06T 15/00 (2011.01); G06T 15/50 (2011.01); G06T 7/50 (2017.01); H04N 5/232 (2006.01); G06T 15/08 (2011.01); H04N 13/00 (2018.01)
CPC H04N 13/111 (2018.05) [G06T 7/0002 (2013.01); G06T 7/50 (2017.01); G06T 7/514 (2017.01); G06T 7/557 (2017.01); G06T 7/579 (2017.01); G06T 7/70 (2017.01); G06T 15/005 (2013.01); G06T 15/08 (2013.01); G06T 15/20 (2013.01); G06T 15/205 (2013.01); G06T 15/503 (2013.01); H04N 5/23222 (2013.01); H04N 5/232945 (2018.08); H04N 13/271 (2018.05); G06T 2207/10052 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30244 (2013.01); G06T 2215/16 (2013.01); H04N 2013/0081 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
rendering a plurality of target viewpoint images based on plurality of multiplane images of a three-dimensional scene, each of the plurality of multiplane images corresponding with a respective one of the plurality of target viewpoint images, each of the plurality of multiplane images associated with a respective one of a plurality of single plane images of the three-dimensional scene, each of the plurality of single plane images being captured from a respective viewpoint, each of the plurality of multiplane images including a respective plurality of depth planes, each of the respective plurality of depth planes including a respective plurality of pixels from the respective single plane image, each of the respective plurality of pixels in the respective plurality of depth planes being positioned at approximately a same distance from the respective viewpoint;
determining a weighted combination of the respective target viewpoint image for each of the plurality of multiplane images via a processor at a computing device, wherein a sampling density of the plurality of single plane images is sufficiently high that the weighted combination satisfies an inequality wherein a maximum pixel disparity of any scene point between adjacent ones of the plurality of target viewpoint images is less than or equal to a minimum of: (a) a number of depth layers associated with the plurality of multiplane images and (b) half of a target rendering resolution for a novel viewpoint image; and
transmitting the weighted combination of the respective target viewpoint image for each of the plurality of multiplane images as the novel viewpoint image.