US 12,333,778 B2
Method and system for keypoint detection based on neural networks
Csaba Gor, Budapest (HU); Peter Rohoska, Budapest (HU); and Andras Kalapos, Budapest (HU)
Assigned to Continental Autonomous Mobility Germany GmbH, Ingolstadt (DE)
Appl. No. 17/996,894
Filed by Continental Autonomous Mobility Germany GmbH, Ingolstadt (DE)
PCT Filed Mar. 17, 2021, PCT No. PCT/EP2021/056887
§ 371(c)(1), (2) Date Oct. 21, 2022,
PCT Pub. No. WO2021/213742, PCT Pub. Date Oct. 28, 2021.
Claims priority of application No. 20170909 (EP), filed on Apr. 22, 2020.
Prior Publication US 2023/0162472 A1, May 25, 2023
Int. Cl. G06V 10/46 (2022.01); G06V 10/764 (2022.01); G06V 10/82 (2022.01); G06V 20/59 (2022.01)
CPC G06V 10/462 (2022.01) [G06V 10/764 (2022.01); G06V 10/82 (2022.01); G06V 20/59 (2022.01)] 14 Claims
OG exemplary drawing
 
1. A computer-implemented method for determining keypoints of multiple objects included in an image and associating the keypoints to the respective objects based on a neural network and a post processing system coupled with the neural network, the method comprising:
providing confidence heatmaps by a neural network, the confidence heatmaps comprising a lower resolution than the image and therefore provide information regarding a rough location of keypoints of interest;
providing refinement offset vectors by the neural network, each refinement offset vector being associated with a certain keypoint, wherein each refinement offset vector provides information for refining the location of the associated keypoint;
providing:
a set of centroid offset vectors by the neural network, each centroid offset vector of the set of centroid offset vectors being associated with a keypoint, the centroid offset vector providing information regarding a distance and direction between the keypoint and a centroid of the object to which the keypoint belongs; or
one or more centroid confidence heatmaps by the neural network, each centroid confidence heatmap comprising a lower resolution than the image and therefore provide information regarding a rough location of a centroid, wherein one or more centroid refinement offset vectors are provided by the neural network, each centroid refinement offset vector being associated with a centroid and providing information for refining a location of the associated centroid;
determining a location of maximum value in the respective confidence heatmaps, using the location of maximum value in the respective confidence heatmaps as rough keypoint locations and refining the rough keypoint locations by adding the refinement offset vectors, thereby obtaining refined keypoint locations;
determining the centroids of the multiple objects by
adding the centroid offset vectors to the refined keypoint locations or the rough keypoint locations; or
refining the rough centroid locations provided by the centroid confidence heatmaps by adding a centroid refinement offset vector to the respective rough centroid location; and
associating keypoints to the objects based on the determined centroids of the objects.