US 11,699,108 B2
Techniques for deriving and/or leveraging application-centric model metric
Arnold Boedihardjo, Westminster, CO (US); Adam Estrada, Westminster, CO (US); Andrew Jenkins, Westminster, VA (US); Nathan Clement, Westminster, CO (US); and Alan Schoen, Westminster, CO (US)
Assigned to MAXAR MISSION SOLUTIONS INC., Westminster, CO (US)
Filed by MAXAR MISSION SOLUTIONS INC., Westminster, CO (US)
Filed on May 29, 2020, as Appl. No. 16/887,670.
Claims priority of provisional application 62/855,138, filed on May 31, 2019.
Prior Publication US 2020/0380307 A1, Dec. 3, 2020
Int. Cl. G06N 20/20 (2019.01); G06N 20/00 (2019.01); G06N 5/04 (2023.01); G06F 18/21 (2023.01); G06F 18/214 (2023.01); G06V 10/75 (2022.01); G06V 10/764 (2022.01); G06V 10/77 (2022.01); G06V 40/16 (2022.01)
CPC G06N 20/20 (2019.01) [G06F 18/214 (2023.01); G06F 18/217 (2023.01); G06N 5/04 (2013.01); G06N 20/00 (2019.01); G06V 10/76 (2022.01); G06V 10/764 (2022.01); G06V 10/7715 (2022.01); G06V 40/172 (2022.01)] 35 Claims
OG exemplary drawing
 
1. A method of quantifying accuracy of a prediction model that has been trained on a data set parameterized by a plurality of features, the model performing in accordance with a theoretical performance manifold over an intractable input space in connection with the features, the method comprising:
determining which of the plurality of features are strongly correlated with performance of the model;
based on the features determined to be strongly correlated with performance of the model, creating a plurality of parameterized sub-models that, in aggregate, approximate the intractable input space;
generating prototype exemplars for each of the created sub-models, the prototype exemplars for each created sub-model being objects to which the model can be applied to result in a match with the respective sub-model; and
quantifying the accuracy of the model using the generated prototype exemplars;
wherein the quantifying of the accuracy of the model further comprises, provided that the prototype exemplars are representative of the intractable input space, approximating the expected performance of the model on a new data set as the sum of a set of one or more performance metrics for the model on each of the prototype exemplars multiplied by the probability of the respective prototype exemplar matching its respective sub-model.