Scoring metric for machine learning method

Question

For a machine learning method X (Deep Neural Nets variant), which performs classification tasks.

In the output layer, for every label method, X also emits the uncertainty U associated with the respective prediction.

So, in case the testing dataset is very different then the predictions made by X should have a high U. (Mean of U across all labels should be high)

In the case of testing, the dataset is indeed from the same source from which training dataset was generated. For true positives, the mean of uncertainty for the correct label would be low. In the case of false positives, the mean of U would be high (ideally).

What is the principled approach to measuring the effectiveness of method X in outputting the uncertainty? In other words, what would be a good metric to understand the performance of the method in computing the uncertainty for each label?

Metric similar to Root mean squared error or others.

Answer 1

To talk about the uncertainty in our model's prediction, we need to adopt a Bayesian framework. Here, our predictions $y^*$ are the mean of the posterior predictive distribution $\hat{y}^* = \mathbb{E}[y^* | y]$, which minimize the mean square error. However, even if we have a value close to 0 or 1 in the classification setting, this does not imply that we necessarily have high confidence in our prediction. To get a sense of the "spread" of possible predictions on average, we can estimate the predictive variance. The link OP provided details this as, $$\underbrace{\tau^{-1}I}_{\text{prior variance}} + \underbrace{\frac{1}{T} \sum_{t=1}^T \hat{y}_t^*(x^*)^T \hat{y}_t^*(x^*) - \mathbb{E}[y^*]^T \mathbb{E}[y^*].}_{\text{variance in predictions}}$$

In this setting, predictions are obtained by averaging over the $T$ drop-outs rather than the single mean from the posterior predictive, $$\mathbb{E}[y^*] = \sum_{t=1}^T \hat{y}_t^*(x^*).$$

Full disclosure, I have not completely read through the author's dissertation, but after skimming it, it appears to link several interesting concepts: 1) Bayesian variational inference and drop-out, and 2) deep neural networks and gaussian processes. IMHO this is the first step to couching neural networks in a proper statistical setting that enables researchers to better understand what is going on and why things work at all.

Answer 2

There are at least two possible interpretations of uncertainty: (a) uncertainty/confidence in the label that the network predicts; (b) uncertainty in the probability/confidence score that the network outputs for the predicted label. (There might be others as well.) Let me explain both of those.

Confidence scores: uncertainty in the label

Standard deep learning methods already provide a confidence score for their prediction. They output not a single label, but a probability distribution on labels (through a final softmax layer). Then a cross-entropy loss function is used to measure how well the predicted distribution matches the known labels, on the training set.

Once you have this, you can easily measure uncertainty. When applied to a test instance $x$, the output will be a probability distribution on labels. If the label $\ell$ has the highest probability (in this distribution), say probability $p$, then $p$ is a measure of the network's uncertainty/confidence: the smaller $p$ is, the more uncertain (the less confident) the network is in its prediction.

For example, suppose we have a neural network whose input is an image of a stoplight, and classifies whether it is showing a red light, i.e., the two labels are "red" or "not red". Such a neural network (if it ends with a softmax layer) actually outputs a vector $(p_\text{red},p_\text{not red})$ where we interpret $p_\text{red}$ as the network's estimate of the probability that the light is red. Thus, the network classifies the image as "red" if $p_\text{red} \ge p_\text{not red}$ and "not red" otherwise, and the confidence score is the probability score associated with that label.

This provides a notion of uncertainty. For instance, suppose the network outputs the vector $(0.8,0.2)$. Then we'll classify the image as "red", and the confidence score indicates that the network thinks there is a 80% chance that the image is red, so that is a notion of how certain it is that the image is "red". This confidence score is also known as a "posterior probability", in a Bayesian framework, and it can be treated as measuring one kind of confidence.

Uncertainty in the confidence score

Alternatively, perhaps what we want to measure is not uncertainty in the predicted label, but uncertainty in the confidence score associated with that label. For instance, if the output vector is $(0.8,0.2)$, then we'll classify the image as "red", with 80% confidence; perhaps what we care about is, how certain/uncertain are we, in the number "80%"?

That can be measured by the variance of this score. We can construct 1000 different neural networks for this task. Each one will output an output vector, and in particular, a confidence score that the image is red. If we want our best estimate of the posterior probability that the image is red, we take the average of those 1000 scores. If we want to know the uncertainty in that estimate, we take the variance (standard deviation) of those 1000 scores. Thus, this might let us say "we classify the image as 80% likely to be 'red', with standard deviation of 5% around the 80%". That variance (or standard deviation) can be taken as a measure of our uncertainty in the number 80%.

This is what Nicolas Mancuso's excellent answer is explaining, in mathematical terms. I've tried to explain it in non-mathematical terms, to help understand the idea.

Why 1000 networks? The number 1000 is arbitrary. A smaller number will be faster. A larger number will provide a more precise measure of the standard deviation.

How do you train 1000 random networks? There are several possibilities. One is to use bagging. Another is to train on the same training set, but use different random values when creating the network (e.g., initialize each network randomly and independently; use dropout, and use different randomization for each network's dropout, and maybe enable dropout at test time too).