US 12,230,007 B2
Training method for multi-object tracking model and multi-object tracking method
Jiawei He, Beijing (CN); Zehao Huang, Beijing (CN); and Naiyan Wang, Beijing (CN)
Assigned to BEIJING TUSEN ZHITU TECHNOLOGY CO., LTD., Beijing (CN)
Filed by Beijing Tusen Zhitu Technology Co., Ltd., Beijing (CN)
Filed on Jan. 31, 2022, as Appl. No. 17/649,511.
Claims priority of application No. 202110130055.4 (CN), filed on Jan. 29, 2021.
Prior Publication US 2022/0245924 A1, Aug. 4, 2022
Int. Cl. G06K 9/00 (2022.01); G06T 7/246 (2017.01); G06V 10/20 (2022.01); G06V 10/62 (2022.01)
CPC G06V 10/62 (2022.01) [G06T 7/246 (2017.01); G06V 10/255 (2022.01); G06T 2207/10016 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A multi-object tracking method,
wherein the multi-object tracking method is performed by a processor configured to implement a trained multi-object tracking neural network model including an object graph extraction network, a tracklet graph extraction network, and a graph matching network, the multi-object tracking method comprising:
constructing, by the processor, based on the object graph extraction network of the trained multi-object tracking neural network model, an object graph according to objects to be tracked in a current video frame, wherein the object graph comprises first vertexes and first edges connecting the first vertexes, the first vertexes of the object graph correspond to the objects to be tracked, and edge features of the first edges between two first vertexes comprise an attribute relationship between the two first vertexes;
constructing, by the processor, based on the tracklet graph extraction network of the trained multi-object tracking neural network model, a tracklet graph according to tracked tracklets of tracked objects in historical video frames, wherein the historical video frames comprise at least two historical frames, wherein the tracklet graph comprises second vertexes and second edges connecting the second vertexes, the second vertexes of the tracklet graph corresponds to the tracked tracklets of the tracked objects in the historical video frames, each of the tracked objects in the historical video frames corresponds to a tracked tracklet in the tracklet graph, and edge features of each second edge between two second vertexes comprise an attribute relationship between the two second vertexes;
performing, by the processor, based on the graph matching network of the trained multi-object tracking neural network model, graph matching on the object graph and the tracklet graph to calculate matching scores between the objects to be tracked and a tracked tracklet in the tracklet graph; and
determining, by the processor, based on the graph matching network of the trained multi-object tracking neural network model, matched tracklets of the objects to be tracked according to the matching scores;
wherein each of the first vertexes has first vertex features, each of the second vertexes has second vertex features, and the performing graph matching on the object graph and the tracklet graph comprises:
for each of the first vertexes in the object graph:
calculating a feature similarity between the first vertex features of the first vertex in the object graph and the second vertex features of each of the second vertexes in the tracklet graph;
obtaining an enhanced vertex feature of the first vertex by weighting and fusing the second vertex features of the second vertexes in the tracklet graph into the first vertex features of the first vertex based on the feature similarity; and
performing graph matching on the object graph and the tracklet graph based on the enhanced vertex feature of the first vertex.