1
$\begingroup$

I have a question about the use of calibration in neural networks and its relation to accuracy. In Guo et al. it says: "Because the parameter T does not change the maximum of the softmax function, the class prediction remains unchanged. In other words, temperature scaling does not affect the model’s accuracy."

This makes sense to me, but then I don't understand why it is used as a method for uncertainty estimation. Suppose we use a softmax function to determine our probabilities, which is known to be bad to measure uncertainty. Now, if we use temperature scaling or something similar for calibration, do the probabilities calculated with the softmax function become true confidence measures? If this were the case, I don't see why this would be necessary, since the accuracy of the model should not be affected by the calibration.

I think I misunderstood something in the procedure and would be very glad if someone could help me!

$\endgroup$

1 Answer 1

1
$\begingroup$

It doesn't change the model's prediction (which class it predicts), but it does change the model's confidence in its prediction (the value of the corresponding softmax output). The point of uncertain estimation is to make this confidence reflect the degree of certainty we should have in the model's prediction.

There is no standard definition of "true confidence measure". The outputs can be viewed as confidences. We can then assess how well calibrated they are. Empirically, temperature scaling typically helps improve how well calibrated these outputs are.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.