US 12,205,225 B2
Appearance capture under ambient lighting conditions using a deep-learning neural network model
Abhijeet Ghosh, Orpington (GB); Gaurav Chawla, Shenfield (GB); Yiming Lin, London (GB); Gilles Rainer, London (GB); Lewis Bridgeman, Guildford (GB); Tristan Wride, Marlow (GB); and Arvin Lin, London (GB)
Assigned to Lumirithmic Limited, Winchester (GB)
Filed by Lumirithmic Limited, Winchester (GB)
Filed on Oct. 24, 2022, as Appl. No. 17/972,009.
Prior Publication US 2024/0135645 A1, Apr. 25, 2024
Prior Publication US 2024/0233265 A9, Jul. 11, 2024
Int. Cl. G06T 17/20 (2006.01); G06T 5/20 (2006.01); G06T 5/70 (2024.01); G06T 15/50 (2011.01); H04N 13/239 (2018.01)
CPC G06T 17/20 (2013.01) [G06T 5/20 (2013.01); G06T 5/70 (2024.01); G06T 15/506 (2013.01); G06T 2207/20021 (2013.01); G06T 2210/44 (2013.01); H04N 13/239 (2018.05)] 18 Claims
OG exemplary drawing
 
1. A method comprising:
receiving a plurality of object images of an object, each object image obtained under ambient lighting conditions using a handheld device and corresponding to a different view direction, wherein the plurality of object images comprises first and second object images corresponding to first and second directions;
determining a global coordinate space based on plurality of object images, and mapping out the relative positions and viewpoints corresponding to each object image;
determining a mesh corresponding to the target region of the object surface based on a first subset of the plurality of object images which comprises two or more object images of the plurality of object images;
determining diffuse and specular maps corresponding to the target region of the object surface based on processing a second subset of the object images using a deep learning neural network model trained to estimate diffuse and specular albedo components based on an input image, wherein the second subset comprises at least the first and second object images;
determining a tangent normal map corresponding to the target region of the object surface based on high-pass filtering each object image of the second subset;
storing and/or outputting the mesh, the diffuse map, the specular map and the tangent normal map;
wherein determining diffuse and specular maps corresponding to the target region of the object surface comprises:
for each of the second subset of the object images:
providing that object image as input to the deep learning neural network model and obtaining a corresponding camera-space diffuse map and a corresponding camera-space specular map as output.