US 12,002,146 B2
3D modeling based on neural light field
Zeng Huang, Los Angeles, CA (US); Jian Ren, Marina Del Ray, CA (US); Sergey Tulyakov, Marina del Rey, CA (US); Menglei Chai, Los Angeles, CA (US); Kyle Olszewski, Los Angeles, CA (US); and Huan Wang, Somerville, MA (US)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Snap Inc., Santa Monica, CA (US)
Filed on Mar. 28, 2022, as Appl. No. 17/656,778.
Prior Publication US 2023/0306675 A1, Sep. 28, 2023
Int. Cl. G06T 15/06 (2011.01); G06T 7/00 (2017.01)
CPC G06T 15/06 (2013.01) [G06T 7/97 (2017.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
receiving, by one or more processors, a set of two-dimensional (2D) images representing a first view of a real-world environment;
applying, by the one or more processors, a machine learning model comprising a neural light field network to the set of 2D images to predict pixel values of a target image representing a second view of the real-world environment, the machine learning model being trained to map a ray origin and direction directly to a given pixel value, the pixel values of the target image being predicted without processing multiple points along a camera ray directed to the given pixel value; and
generating, by the one or more processors, a three-dimensional (3D) model of the real-world environment based on the set of 2D images and the predicted target image.