I am fine-tuning the inception-v3 network(pretrained on ILSVRC) of tensorflow in a binary classification problem. I want to recognize if an image contains food or not. My dataset is imbalanced, the non-food class contains 5 times more images than the food class. I read that in an imbalanced dataset, I should train the network with balanced batches to get right results. So I split the dominant class in 5 splits and when I have to train the network for 50000 iterations I am changing the split every 10000 iterations, so that the batches that are passed to the training are balanced with food and non-food. I am testing the network in a balanced test-set. From the first 5000 iterations I am getting high accuracy in the food images, but the non-food class gives me imbalanced accuracy. I can receive high accuracy in the beginning and then it will be low. By accuracy I mean (number of rightly predicted food) / (number of food) for the food class and respectively for the non-food. Generally the network learns the food characteristics because the softmax can output non-food for a food image, but the localization of food would be right using the weights of food (I made test). To sum up, the network is probably overfitted but I don't know yet. Any ideas of what might be happening?
It's hard to know, from just the information you've given us. It could be any number of things: overfitting, insufficient training data, class imbalance, or something else. The first step is probably to figure out the source of the problem, then you can explore the solutions.
There are standard diagnostics to check for overfitting (cross-validation, comparing accuracy on the validation set to accuracy on the training set, etc.). I'd suggest you use them to diagnose whether the network seems to be overfitting. There are many ways to combat overfitting: regularization, early-stop training, dropout, etc.
There are also ways to check if the problem is that you just don't have enough training data. For instance, you can take different-size subsets of your training set (10K, 20K, 30K, 40K, 50K) and plot how accuracy depends on the size of the training set. If accuracy seems to be gradually improving as training set size increases and doesn't appear to have reached a plateau yet, that's a suggestion that obtaining more training data might be useful.
Finally, class imbalance might be a factor. When you have 5 times as many non-food instances as food instances, that's call class imbalance. There's lots written on that subject and many techniques (undersampling the minority class, oversampling the majority class, using class weights, adjusting the threshold for classification, and more). I suggest you do some reading about that topic. It's not necessarily true that you need to train on a training set with balanced classes; that's one strategy, but it's not the only one, and it may not be optimal.
Hopefully this will help you narrow down what might be going on and then ask a more focused question.