Amount of classes for a semantic segmentation neural network

Question

I am currently working on implementation of semantic segmentation of images neural network, and try to implement one of the already existing solution such as Fully Convolutional Neural Network 1.

Data that I am using is based on Pascal-Context dataset [2], which has additional labeling to original 20-class PASCAL VOC dataset. This results in a dataset with over 450-classes.

Problem

Initial 20-classes do not not match classes that I would like to achieve for indoor scenes. Therefore, I have created a short list of 12-classes that I would like to capture, which are in Pascal Context 450-classes dataset.

I managed to convert the data and now trying to start training. I am following this tutorial [3] on Matlab, which provides an example of an Image with classes overlay and all pixels colored.

However, in my scenario, I only want to be able to distinguish elements such as tvmonitor, sofa, wall and ignoring all other elements, which might be there. Matlab tutorial states: "Areas with no color overlay do not have pixel labels and are not used during training."[

As you can see above, I have two classes present in the picture, but I am not sure whether I should also include class Background, which would put overlay on everything that is not within my list of classes, and include that as an additional class in my training or not.

In summary, I am wondering whether Background needs to be provided as an additional class to my list of the classes that I would like to classify or not, even if this background class usually takes majority of each of images. Would that result in everything being classified as background?

References:

1 https://github.com/shelhamer/fcn.berkeleyvision.org

[2] https://www.cs.stanford.edu/~roozbeh/pascal-context/

[3] https://uk.mathworks.com/help/vision/examples/semantic-segmentation-using-deep-learning.html#d119e321

Answer 1

It's hard to tell whether you are planning on doing image classification or object recognition/detection.

If you're doing image classification, you need each image to have only a single object, and the object needs to be in approximately the same position every time. You train a neural network to output the label (which object it is).

If you're doing object recognition/detection, in the training set it's not enough to have a label for each image. Rather, for each image you should have a list of the objects (labels) and the bounding box of each. Then, you train a neural network to output the label (which object) and location (bounding box) of the object. That requires a different architecture than object classification.

I suggest you spend some time reading up on object recognition/detection and how to do that with a neural network, since it sounds like you might not be familiar with that yet.

As far as the background class: I'm assuming you mean a "none-of-the-above" class. If you are doing image classification and you expect to have some images that aren't of any of the 12 labels, then yes, you should have a 13th label for "none of the above" (i.e., background), and your training set should contain examples of all 13 classes. If you're doing object recognition/detection, that isn't necessary; instead, typically the training set needs to have realistic images that contain zero, one, or more objects, though that might depend on the specific architecture and approach for object recognition/detection you are using.