US 12,066,282 B2
Volumetric performance capture with relighting
Sean Ryan Francesco Fanello, San Francisco, CA (US); Kaiwen Guo, Beijing (CN); Peter Christopher Lincoln, San Francisco, CA (US); Philip Lindsley Davidson, Arlington, MA (US); Jessica L. Busch, Long Beach, CA (US); Xueming Yu, Arcadia, CA (US); Geoffrey Harvey, Culver City, CA (US); Sergio Orts Escolano, San Francisco, CA (US); Rohit Kumar Pandey, Mountain View, CA (US); Jason Dourgarian, Los Angeles, CA (US); Danhang Tang, San Francisco, CA (US); Adarsh Prakash Murthy Kowdle, San Francisco, CA (US); Emily B. Cooper, San Francisco, CA (US); Mingsong Dou, Cupertino, CA (US); Graham Fyffe, Los Angeles, CA (US); Christoph Rhemann, Marina Del Rey, CA (US); Jonathan James Taylor, San Francisco, CA (US); Shahram Izadi, Tiburon, CA (US); and Paul Ernest Debevec, Culver City, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Appl. No. 17/413,847
Filed by GOOGLE LLC, Mountain View, CA (US)
PCT Filed Nov. 11, 2020, PCT No. PCT/US2020/059973
§ 371(c)(1), (2) Date Jun. 14, 2021,
PCT Pub. No. WO2021/096930, PCT Pub. Date May 20, 2021.
Claims priority of provisional application 62/934,320, filed on Nov. 12, 2019.
Prior Publication US 2022/0065620 A1, Mar. 3, 2022
Int. Cl. G01B 11/25 (2006.01); G01B 11/245 (2006.01); G06T 15/50 (2011.01); G06T 17/20 (2006.01)
CPC G01B 11/2513 (2013.01) [G01B 11/245 (2013.01); G06T 15/506 (2013.01); G06T 17/205 (2013.01)] 29 Claims
OG exemplary drawing
 
1. An apparatus comprising:
a plurality of lights configured to project alternating red-green-blue (RGB) color gradient illumination patterns onto an object or human performer at a predetermined frequency;
a plurality of cameras configured to capture RGB images of an object or human performer corresponding to the alternating RGB color gradient illumination patterns;
a plurality of depth sensors to capture a depth map of the object or human performer at the predetermined frequency; and
at least one processor that implements a machine learning algorithm to produce a three-dimensional (3D) model of the object or human performer based on the captured RGB images and the depth map, the 3D model comprising relighting parameters used to relight the 3D model under different lighting conditions.