US 12,340,467 B2
Generating three-dimensional object models from two-dimensional images
Dominik Kulon, London (GB); Riza Alp Guler, London (GB); Iason Kokkinos, London (GB); and Stefanos Zafeiriou, London (GB)
Assigned to Snap Inc., Santa Monica, CA (US)
Appl. No. 17/760,424
Filed by Snap Inc., Santa Monica, CA (US)
PCT Filed Feb. 17, 2020, PCT No. PCT/GB2020/050371
§ 371(c)(1), (2) Date Aug. 9, 2022,
PCT Pub. No. WO2021/165628, PCT Pub. Date Aug. 26, 2021.
Prior Publication US 2023/0070008 A1, Mar. 9, 2023
Int. Cl. G06T 17/20 (2006.01); G06T 3/40 (2006.01); G06T 7/00 (2017.01)
CPC G06T 17/20 (2013.01) [G06T 3/40 (2013.01); G06T 7/97 (2017.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A computer implemented method, comprising:
receiving, as input to an embedding neural network that comprises one or more encoder parameters, a two-dimensional training image, wherein the two-dimensional training image comprises an object and is associated with a predetermined three-dimensional model that comprises vertices;
generating, using the embedding neural network, a first embedded representation of the two-dimensional training image;
inputting the first embedded representation into a decoder model, the decoder model comprising a regression model that comprises one or more decoder parameters;
generating, using the decoder model, a generated three-dimensional model of the object based on the first embedded representation, the generated three-dimensional model comprising vertices;
comparing, using a loss function, the generated three-dimensional model of the object to the predetermined three-dimensional model associated with the two-dimensional training image;
updating the one or more encoder parameters and the one or more decoder parameters of the decoder model in dependence on the comparing: receiving, as input to the embedding neural network, a two-dimensional object image, wherein the two-dimensional object image comprises an image of the object;
generating, using the embedding neural network, a second embedded representation of a two-dimensional object image;
inputting the second embedded representation of the two-dimensional object image into the decoder model that comprises the regression model; and
generating, using the decoder model, a resulting three-dimensional model of the object from the second embedded representation, the resulting three-dimensional model comprising a plurality of nodes in a mesh;
wherein the loss function comprises a vertex term and an edge term, the vertex term comprising first differences between vertices of the generated three-dimensional model and vertices of the predetermined three-dimensional model, the edge term comprising second differences between edge lengths of the generated three-dimensional model and edge lengths of the predetermined three-dimensional model.