US 11,710,287 B2
Generative latent textured proxies for object category modeling
Ricardo Martin Brualla, Seattle, WA (US); Daniel Goldman, Seattle, WA (US); Sofien Bouaziz, Los Gatos, CA (US); Rohit Kumar Pandey, Mountain View, CA (US); and Matthew Brown, Seattle, WA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Appl. No. 17/309,817
Filed by GOOGLE LLC, Mountain View, CA (US)
PCT Filed Aug. 4, 2020, PCT No. PCT/US2020/070359
§ 371(c)(1), (2) Date Jun. 21, 2021,
PCT Pub. No. WO2022/005523, PCT Pub. Date Jan. 6, 2022.
Claims priority of provisional application 62/705,500, filed on Jun. 30, 2020.
Prior Publication US 2022/0051485 A1, Feb. 17, 2022
Int. Cl. G06T 19/20 (2011.01); G06T 15/00 (2011.01); G06T 15/04 (2011.01); G06T 15/50 (2011.01); G06V 10/94 (2022.01)
CPC G06T 19/20 (2013.01) [G06T 15/005 (2013.01); G06T 15/04 (2013.01); G06T 15/506 (2013.01); G06V 10/95 (2022.01); G06T 2219/2012 (2013.01); G06T 2219/2021 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method utilizing at least one processing device to perform operations including:
receiving a pose associated with an object in image content;
generating a plurality of three-dimensional (3D) proxy geometries of the object, the plurality of 3D proxy geometries being based on a shape of the object;
generating, based on the plurality of 3D proxy geometries, a plurality of neural textures of the object, each of the plurality of neural textures defining a plurality of different shapes and appearances representing the object, the plurality of neural textures being configured to reconstruct a hidden portion of the object captured in the image content;
providing the plurality of neural textures to a neural renderer, the plurality of neural textures being provided in a stacked formation, the hidden portion being reconstructed based on the stacked formation of the plurality of neural textures;
generating, by the neural renderer, transparent layers of the object and surfaces behind the transparent layers of the object based on the plurality of neural textures;
receiving, from the neural renderer and based on the plurality of neural textures, a color image and an alpha mask representing an opacity of at least a portion of the object; and
generating a composite image based on the pose, the color image, and the alpha mask.