US 12,131,422 B2
Uncertainty-aware fusion towards large-scale NeRF
Bingbing Zhuang, San Jose, CA (US); Samuel Schulter, New York, NY (US); Yi-Hsuan Tsai, Santa Clara, CA (US); Buyu Liu, Cupertino, CA (US); and Nanbo Li, Edinburgh (GB)
Assigned to NEC Corporation, Tokyo (JP)
Filed by NEC Laboratories America, Inc., Princeton, NJ (US)
Filed on Oct. 11, 2022, as Appl. No. 17/963,471.
Claims priority of provisional application 63/278,529, filed on Nov. 12, 2021.
Prior Publication US 2023/0154104 A1, May 18, 2023
Int. Cl. G06T 17/00 (2006.01); G06T 15/00 (2011.01); G06T 15/20 (2011.01); G06V 10/774 (2022.01); G06V 10/82 (2022.01); G06V 20/40 (2022.01)
CPC G06T 15/20 (2013.01) [G06T 15/00 (2013.01); G06T 17/00 (2013.01); G06V 10/774 (2022.01); G06V 10/82 (2022.01); G06V 20/41 (2022.01); G06T 2200/08 (2013.01); G06T 2210/56 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for achieving high-fidelity novel view synthesis and 3D reconstruction for large-scale scenes, the method comprising:
obtaining images from a video stream received from a plurality of video image capturing devices;
grouping the images into different image clusters representing a large-scale 3D scene;
training a neural radiance field (NeRF) and an uncertainty multilayer perceptron (MLP) for each of the image clusters to generate a plurality of NeRFs and a plurality of uncertainty MLPs for the large-scale 3D scene;
applying a rendering loss and an entropy loss to the plurality of NeRFs;
performing uncertainty-based fusion to the plurality of NeRFs to define a fused NeRF, and jointly fine-tuning the plurality of NeRFs and the plurality of uncertainty MLPs; and
during inference, applying the fused NeRF for novel view synthesis of the large-scale 3D scene.