Big number of false positives in binary classification

Question

I am fine-tuning the inception-v3 network(pretrained on ILSVRC) of tensorflow in a binary classification problem. I want to recognize if an image contains food or not. My dataset is imbalanced, the non-food class contains 5 times more images than the food class. I read that in an imbalanced dataset, I should train the network with balanced batches to get right results. So I split the dominant class in 5 splits and when I have to train the network for 50000 iterations I am changing the split every 10000 iterations, so that the batches that are passed to the training are balanced with food and non-food. I am testing the network in a balanced test-set. From the first 5000 iterations I am getting high accuracy in the food images, but the non-food class gives me imbalanced accuracy. I can receive high accuracy in the beginning and then it will be low. By accuracy I mean (number of rightly predicted food) / (number of food) for the food class and respectively for the non-food. Generally the network learns the food characteristics because the softmax can output non-food for a food image, but the localization of food would be right using the weights of food (I made test). To sum up, the network is probably overfitted but I don't know yet. Any ideas of what might be happening?

Answer 1

It's hard to know, from just the information you've given us. It could be any number of things: overfitting, insufficient training data, class imbalance, or something else. The first step is probably to figure out the source of the problem, then you can explore the solutions.

There are standard diagnostics to check for overfitting (cross-validation, comparing accuracy on the validation set to accuracy on the training set, etc.). I'd suggest you use them to diagnose whether the network seems to be overfitting. There are many ways to combat overfitting: regularization, early-stop training, dropout, etc.

There are also ways to check if the problem is that you just don't have enough training data. For instance, you can take different-size subsets of your training set (10K, 20K, 30K, 40K, 50K) and plot how accuracy depends on the size of the training set. If accuracy seems to be gradually improving as training set size increases and doesn't appear to have reached a plateau yet, that's a suggestion that obtaining more training data might be useful.

Finally, class imbalance might be a factor. When you have 5 times as many non-food instances as food instances, that's call class imbalance. There's lots written on that subject and many techniques (undersampling the minority class, oversampling the majority class, using class weights, adjusting the threshold for classification, and more). I suggest you do some reading about that topic. It's not necessarily true that you need to train on a training set with balanced classes; that's one strategy, but it's not the only one, and it may not be optimal.

Hopefully this will help you narrow down what might be going on and then ask a more focused question.

Answer 2

The neural network you're building in Tensorflow does not predict categories. The neural network gives predictions on a continuum. It is then up to you to decide what you want to do with those predictions. Often, the predictions are in the range $[0,1]$ and can be interpreted as probabilities of membership in one of the classes, typically the minority category (but it depends on the exact way you have coded the problem). A common decision rule for those predictions on $[0,1]$ is to predict one category if the prediction is below $0.5$ and the other category otherwise. However, you do not have to use this decision rule. Indeed, receiver-operator characteristic and precision-recall curves exist by varying that threshold and not just using the software default of $0.5$.

There are reasons to evaluate the raw outputs of the model [1, 2], but if you don't want to use the arguably hard-to-interpret measures of performance like log loss and Brier score that are discussed in the links, favoring easy-to-interpret measures like accuracy, you're at least allowed to vary the decision rule and see if your two-stage pipeline of model predictions followed by a classification based on those model predictions yields better classifications than the software-default decision rule. You can even do this without having to worry about the class imbalance giving all majority-class classifications, since you vary the threshold and can have a very low threshold that gives many (even all) minority-class classifications.

You might, for instance, write a loop over all values $\left\{0, 0.001, 0.002,\dots, 0.998, 0.999, 1\right\}$ and plot various classification metrics at the various thresholds in that set of thresholds. My suspicion is that you have gotten poor classification results due to using a software-default threshold of $0.5$ and that you will get better performance with a lower threshold.