I am currently working on implementation of semantic segmentation of images neural network, and try to implement one of the already existing solution such as Fully Convolutional Neural Network 1.

Data that I am using is based on Pascal-Context dataset [2], which has additional labeling to original 20-class PASCAL VOC dataset. This results in a dataset with over 450-classes.


Initial 20-classes do not not match classes that I would like to achieve for indoor scenes. Therefore, I have created a short list of 12-classes that I would like to capture, which are in Pascal Context 450-classes dataset.

I managed to convert the data and now trying to start training. I am following this tutorial [3] on Matlab, which provides an example of an Image with classes overlay and all pixels colored.

However, in my scenario, I only want to be able to distinguish elements such as tvmonitor, sofa, wall and ignoring all other elements, which might be there. Matlab tutorial states: "Areas with no color overlay do not have pixel labels and are not used during training."[D

As you can see above, I have two classes present in the picture, but I am not sure whether I should also include class Background, which would put overlay on everything that is not within my list of classes, and include that as an additional class in my training or not.

In summary, I am wondering whether Background needs to be provided as an additional class to my list of the classes that I would like to classify or not, even if this background class usually takes majority of each of images. Would that result in everything being classified as background?


1 https://github.com/shelhamer/fcn.berkeleyvision.org

[2] https://www.cs.stanford.edu/~roozbeh/pascal-context/

[3] https://uk.mathworks.com/help/vision/examples/semantic-segmentation-using-deep-learning.html#d119e321

  • $\begingroup$ The title you have chosen is not well suited to representing your question. Please take some time to improve it; we have collected some advice here. Thank you! $\endgroup$
    – Raphael
    Jan 28, 2018 at 14:09
  • $\begingroup$ Note that there are also Cross Validated, Artificial Intelligence, and Computational Science. Please don't crosspost, but if you don't get satisfactory answers here, we can migrate. $\endgroup$
    – Raphael
    Jan 28, 2018 at 14:09

1 Answer 1


It's hard to tell whether you are planning on doing image classification or object recognition/detection.

If you're doing image classification, you need each image to have only a single object, and the object needs to be in approximately the same position every time. You train a neural network to output the label (which object it is).

If you're doing object recognition/detection, in the training set it's not enough to have a label for each image. Rather, for each image you should have a list of the objects (labels) and the bounding box of each. Then, you train a neural network to output the label (which object) and location (bounding box) of the object. That requires a different architecture than object classification.

I suggest you spend some time reading up on object recognition/detection and how to do that with a neural network, since it sounds like you might not be familiar with that yet.

As far as the background class: I'm assuming you mean a "none-of-the-above" class. If you are doing image classification and you expect to have some images that aren't of any of the 12 labels, then yes, you should have a 13th label for "none of the above" (i.e., background), and your training set should contain examples of all 13 classes. If you're doing object recognition/detection, that isn't necessary; instead, typically the training set needs to have realistic images that contain zero, one, or more objects, though that might depend on the specific architecture and approach for object recognition/detection you are using.

  • $\begingroup$ I am trying to do object detection a.k.a semantic segmentation ,which produces a label for each pixel within the image, and specifies which class each pixels belongs to. I have read multiple papers, but they do not specify whether they include background or not in their training. $\endgroup$ Jan 29, 2018 at 14:03
  • $\begingroup$ @JohhnyBravo, the easy way to tell is that you need the training set to be on the same distribution on the images you will apply it to. So if you want the system to output "background" ("none-of-the-above") labels on pixels that aren't part of one of the 12 objects... then yes, you'll need those labels on training images, too. $\endgroup$
    – D.W.
    Jan 29, 2018 at 15:21
  • $\begingroup$ Great, so assuming that I have an image with an object that doesn't belong to any of the categories, this would result in the output as none-of-the-above in case of additional "background" class, while in scenario when this class is missing, this would be classified as one of the classes that are provided to the system right? $\endgroup$ Jan 29, 2018 at 15:28
  • 1
    $\begingroup$ If you are doing object detection / semantic segmentation, an image isn't labeled as a class. Rather, as you said above, each pixel is labelled. So, if you have an image with an object that doesn't belong to any of the 12 categories, then the correct label for each pixel of that image would be .... (I am sure that if you think about it you can fill in the blank here.) $\endgroup$
    – D.W.
    Jan 29, 2018 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.