| CPC G06N 3/08 (2013.01) [G06F 17/16 (2013.01); G06F 17/18 (2013.01); G06N 3/047 (2023.01)] | 15 Claims |
|
1. A computer-implemented method of calibrating a trained classification model which is trained to classify input samples according to a plurality of classes and to provide associated prediction probabilities, the trained classification model including a plurality of hidden layers and at least one activation layer, the method comprising the following steps:
accessing the trained classification model;
accessing a plurality of validation samples, each validation sample of the validation samples having a ground-truth label, the ground-truth label indicating a ground-truth class;
applying the trained classification model to the plurality of validation samples;
obtaining, for each validation sample of the validation samples, an output logit vector from a layer of the trained classification model preceding a last activation layer;
training a calibration module for adjusting prediction probabilities, the prediction probabilities being derived from output logit vectors, the calibration module including at least one of a finetuning submodule for adjusting the prediction probabilities by finetuning the output logit vector and a binning submodule for adjusting the prediction probabilities by binning the output logit vector; and
wherein the training the calibration module includes training a finetuning model wherein model parameters of the finetuning model are determined by:
for each validation sample of the plurality of validation samples:
determining a ground-truth probability from the output logit vector, the ground-truth probability being a predicted probability associated with the ground-truth class of the validation sample,
determining an anchor probability from the output logit vector, the anchor probability being a highest probability of an incorrect class, and
determining a prediction difficulty by subtracting the ground-truth probability from the anchor probability;
determining the model parameters of the finetuning model by minimizing a finetuning loss function averaged over the plurality of validation samples, the finetuning loss function comprising a modulation term based on the prediction difficulty for each validation sample, the determined model parameters defining the trained finetuning model; and
storing the trained finetuning model in the calibration module; and
appending the trained calibration module to the trained classification model to obtain a calibrated classification model.
|