US 11,869,132 B2
Neural network based 3D object surface mapping
Vladimir Kim, Seattle, WA (US); Noam Aigerman, San Francisco, CA (US); Niloy J. Mitra, Potters Bar (GB); and Luca Morreale, London (GB)
Assigned to Adobe Inc., San Jose, CA (US); and UCL Business Ltd., London (GB)
Filed by Adobe Inc., San Jose, CA (US); and UCL Business Ltd., London (GB)
Filed on Nov. 29, 2021, as Appl. No. 17/537,343.
Prior Publication US 2023/0169714 A1, Jun. 1, 2023
Int. Cl. G06T 17/20 (2006.01); G06T 15/04 (2011.01); G06N 3/08 (2023.01); G06N 3/045 (2023.01)
CPC G06T 15/04 (2013.01) [G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06T 17/20 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
generating a surface mapping function for mapping a first surface of a first three-dimensional (3D) object in a 3D space to a second surface of a second 3D object in the 3D space, wherein the surface mapping function is defined by a first representation of the first surface, a second representation of the second surface, and a neural network model configured as an intermediary between the first representation and the second representation to map a first two-dimensional (2D) representation to a second 2D representation;
selecting a set of points in the first 2D representation, the set of points comprising vertex points and internal points of a first 3D mesh, wherein:
the first representation of the first surface is produced by minimizing a loss function defined based on Euclidean distances between (a) output points of the neural network model using the set of points as input and (b) ground-truth points of the set of points in the 3D space, so that the first representation of the first surface corresponds to a mapping from the first 2D representation of the first surface to the first surface of the first 3D object;
the second representation corresponds to a mapping from the second 2D representation of the second surface to the second surface of the second 3D object; and
generating the surface mapping function comprises adjusting parameters of the neural network model to optimize an objective function, wherein the objective function comprises a distortion term defining distortion between the first surface and the second surface mapped through the surface mapping function;
applying a feature of the first 3D mesh on the first surface to a second 3D mesh on the second surface to produce a modified second surface, wherein the first 3D mesh on the first surface maps to the second 3D mesh on the second surface as determined by the surface mapping function; and
rendering, by a rendering module, the modified second surface.