US 12,299,920 B2
Method for detecting hand key points, method for recognizing gesture, and related devices
Wei Xiang, Guangzhou (CN); and Yifeng Wang, Guangzhou (CN)
Assigned to BIGO TECHNOLOGY PTE. LTD., Singapore (SG)
Appl. No. 17/780,694
Filed by BIGO TECHNOLOGY PTE. LTD., Singapore (SG)
PCT Filed Aug. 7, 2020, PCT No. PCT/CN2020/107960
§ 371(c)(1), (2) Date May 27, 2022,
PCT Pub. No. WO2021/103648, PCT Pub. Date Jun. 3, 2021.
Claims priority of application No. 201911198688.8 (CN), filed on Nov. 29, 2019.
Prior Publication US 2023/0252670 A1, Aug. 10, 2023
Int. Cl. G06T 7/73 (2017.01); G06V 40/10 (2022.01); G06V 40/20 (2022.01)
CPC G06T 7/73 (2017.01) [G06V 40/11 (2022.01); G06V 40/20 (2022.01); G06T 2207/20076 (2013.01); G06T 2207/30196 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A method for detecting hand key points, comprising:
acquiring a hand image to be detected;
acquiring heat maps of the hand key points by inputting the hand image into a pre-trained heat map model, wherein the heat maps comprise two-dimensional coordinates of the hand key points;
acquiring hand structured connection information by inputting the heat maps and the hand image into a pre-trained three-dimensional information prediction model;
determining, based on the hand structured connection information and the two-dimensional coordinates in the heat maps, three-dimensional coordinates of the hand key points in a world coordinate system; and
recognizing, based on three-dimensional coordinates of the hand key points in the world coordinate system, a gesture expressed by a hand in the hand image to detect the hand key points in real time through a mobile terminal;
wherein the hand structured connection information comprises joint bending angles formed by the hand key points and Euler angles of the hand; and determining, based on the hand structured connection information and the two-dimensional coordinates in the heat maps, the three-dimensional coordinates of the hand key points in the world coordinate system comprises:
calculating, based on the joint bending angles, a first direction vector of a vector formed by two hand key points in a hand coordinate system of the hand;
converting the first direction vector into a second direction vector in the world coordinate system based on the Euler angles;
calculating a vector length of the vector based on the two-dimensional coordinates in the heat maps;
acquiring the vector by calculating a product of the vector length and the second direction vector; and
calculating the three-dimensional coordinates of the two hand key points forming the vector in the world coordinate system based on the vector.