US 12,020,688 B1
Evaluation method and device of air traffic control speech recognition system
Weijun Pan, Guanghan (CN); Peiyuan Jiang, Guanghan (CN); Qinghai Zuo, Guanghan (CN); Xuan Wang, Guanghan (CN); Rundong Wang, Guanghan (CN); Tian Luan, Guanghan (CN); Zixuan Wang, Guanghan (CN); Jian Zhang, Guanghan (CN); Qianlan Jiang, Guanghan (CN); and Yidi Wang, Guanghan (CN)
Assigned to CIVIL AVIATION FLIGHT UNIVERSITY OF CHINA, Guanghan (CN)
Filed by CIVIL AVIATION FLIGHT UNIVERSITY OF CHINA, Guanghan (CN)
Filed on Dec. 26, 2023, as Appl. No. 18/395,770.
Claims priority of application No. 202211687813.3 (CN), filed on Dec. 28, 2022.
Int. Cl. G10L 15/01 (2013.01); G10L 15/02 (2006.01); G08G 5/00 (2006.01)
CPC G10L 15/01 (2013.01) [G10L 15/02 (2013.01); G08G 5/0013 (2013.01)] 7 Claims
OG exemplary drawing
 
1. An evaluation method of an air traffic control (ATC) speech recognition system, comprising:
S1, obtaining ATC speech data, grading, based on a quality level of the ATC speech data, the ATC speech data to obtain ATC speech data with different quality level, and inputting the ATC speech data with different quality level into a to-be-evaluated ATC speech recognition system;
S2, constructing an evaluation index system; wherein indexes of the evaluation index system comprise utility layer indexes and support layer indexes;
S3, determining, by using an improved group-G2 method and an improved criteria importance through intercriteria correlation (CRITIC) method, weights of the utility layer indexes and weights of the support layer indexes under the ATC speech data with different quality levels;
S4, calculating, according to the weights of the utility layer indexes and the weights of the support layer indexes under the ATC speech data with different quality levels, scores of a utility layer and scores of a support layer under the ATC speech data with different quality levels by using an improved technique for order preference by similarity to ideal solution (TOPSIS) method; and adding the scores of the utility layer to obtain a comprehensive score of the utility layer, and adding the scores of the support layer to obtain a comprehensive score of the support layer;
wherein the TOPSIS method is to determine Z+ and Z− of an index through a statistical method, to thereby score a recognition performance of the to-be-evaluated ATC speech recognition system; and
wherein Z+ represents a set containing optimal values of evaluation indexes of the to-be-evaluated ATC speech recognition system, and Z− represents a set containing worst values of the evaluation indexes of the to-be-evaluated ATC speech recognition system;
S5, determining, according to the comprehensive score of the utility layer and the comprehensive score of the support layer, a weight of the utility layer and a weight of the support layer of the to-be-evaluated ATC speech recognition system by using the improved group-G2 method and the improved CRITIC method;
S6, multiplying the weight of the utility layer and the comprehensive score of the utility layer to obtain a first product, multiplying the weight of the support layer and the comprehensive score of the support layer to obtain a second product, and adding the first product and the second product to obtain a comprehensive score of the to-be-evaluated ATC speech recognition system; and
S7, determining, according to the comprehensive score of the to-be-evaluated ATC speech recognition system, a level of the recognition performance of the to-be-evaluated ATC speech recognition system.