US 12,111,818 B2
Comparing ordered sets generated by different data-processing engines
Peter Joseph Rise, Bellevue, WA (US)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on Dec. 13, 2022, as Appl. No. 18/080,484.
Prior Publication US 2024/0193150 A1, Jun. 13, 2024
Int. Cl. G06F 7/00 (2006.01); G01C 21/00 (2006.01); G01C 21/34 (2006.01); G06F 16/23 (2019.01); G06F 16/2457 (2019.01); G06F 16/29 (2019.01); G06F 17/00 (2019.01)
CPC G06F 16/2365 (2019.01) [G01C 21/3415 (2013.01); G01C 21/3811 (2020.08); G01C 21/3815 (2020.08); G01C 21/3859 (2020.08); G06F 16/24578 (2019.01); G06F 16/29 (2019.01)] 19 Claims
OG exemplary drawing
 
1. A computer-implemented method for updating data-processing functionality, comprising:
obtaining a first ordered set of first-set result items produced by a first data-processing engine in response to an input query;
obtaining a second ordered set of second-set result items produced by a second data-processing engine in response to the same input query, the second data-processing engine serving as a reference engine that provides ground-truth result items;
generating accuracy information by assessing accuracy of the first-set result items relative to the second-set result items,
the generating being based on an assessment of item similarity between individual pairings formed from the first ordered set and the second ordered set, and based on a comparison of an order of the first-set result items and an order of the second-set result items; and
producing output information that identifies deficiencies in the first data-processing engine based on the accuracy information, which enables modification of a configuration of the first data-processing engine to increase quality of result items produced by the first data-processing engine in a production stage of operation,
the generating including:
producing matching information that uses a prescribed test of similarity to identify first-set result items that match respective second-set result items; and
determining relevance information based on the matching information, the relevance information including individual relevance measures that express relevance of respective first-set result items in the first ordered set,
wherein a particular first-set result item matches a particular second-set result item, and wherein the determining relevance information assigns a relevance measure to the particular first-set result item based on a position of the particular second-set result item in the second ordered set.