US 11,941,772 B2
Method and system for generating a 3D reconstruction of a human
Fabio Cappello, London (GB); Nigel John Williams, London (GB); and Rajeev Gupta, London (GB)
Assigned to Sony Interactive Entertainment Inc., Tokyo (JP)
Appl. No. 17/273,573
Filed by Sony Interactive Entertainment Inc., Tokyo (JP)
PCT Filed Sep. 4, 2019, PCT No. PCT/GB2019/052457
§ 371(c)(1), (2) Date Mar. 4, 2021,
PCT Pub. No. WO2020/053551, PCT Pub. Date Mar. 19, 2020.
Claims priority of application No. 1814835 (GB), filed on Sep. 12, 2018.
Prior Publication US 2021/0256776 A1, Aug. 19, 2021
Int. Cl. G06T 19/20 (2011.01); G06N 3/04 (2023.01); G06N 3/08 (2023.01); G06T 7/10 (2017.01); G06T 7/50 (2017.01); G06T 7/90 (2017.01); G06T 15/04 (2011.01); G06T 17/00 (2006.01)
CPC G06T 19/20 (2013.01) [G06N 3/04 (2013.01); G06N 3/08 (2013.01); G06T 7/10 (2017.01); G06T 7/50 (2017.01); G06T 7/90 (2017.01); G06T 15/04 (2013.01); G06T 17/00 (2013.01); G06T 2200/08 (2013.01); G06T 2207/10024 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30196 (2013.01); G06T 2219/2021 (2013.01)] 15 Claims
OG exemplary drawing
 
1. A method of generating a three-dimensional (3D) reconstruction of a human, the method comprising:
obtaining at least one colour image and corresponding depth image of a scene, the at least one colour image and corresponding depth image comprising a human subject that is at least partially occluded by one or more items;
identifying in the at least one colour image, regions corresponding to the non-occluded parts of the human subject;
identifying in the at least one colour image, regions corresponding to one or more items occluding the human subject;
generating a point cloud of the scene based on the at least one colour image and corresponding depth image of the scene, the point cloud comprising regions corresponding to the regions identified in the at least one colour image as corresponding to the non-occluded parts of the human subject;
removing, from the point cloud, points corresponding to the one or more items identified as occluding the human subject to thereby obtain a modified point cloud for which the points corresponding to the one or more items identified as occluding the human subject have been removed;
adjusting one or more parameters of a parametric model based on the modified point cloud, the adjusted parametric model providing an estimate of points in the point cloud corresponding to the at least partially occluded parts of the human subject, wherein the parametric model defines a 3D parametrised shape of a human and wherein adjusting the one or more parameters is such that the model is morphed so as to more closely correspond to the human subject; and
generating a 3D reconstruction of the human subject based on the adjusted parametric model.