How deep do neural networks need to be?

Question

My question is a bit on the philosophical side, and there is probably not one single 'correct' answer on this. Nonetheless, I'm curious to hear your opinion...

I'm currently designing a convolutional neural network to locate features in images taken by a microscope. These features are geometrically quite simple and similar to one another, with only minor contextual variations from image to image. When I look at modern general-purpose covnets, they often have tens of convolution/pooling layers, making them a bit heavy. I can imagine that one needs many layers of abstraction when detecting cars, cats, and cods all in once, which are very distinct from one another.

Returning now to my more specialist network: would it need as many conv/pool layers as a general-purpose covnet? Or should I consider a different strategy all together (like having more fully-connected nodes)?

EDIT: Based on the comments, I feel I should rephrase/clarify my question a bit. Recently I looked into the YOLO architecture, which seems to strike a nice balance between accuracy and speed. The original version of YOLO features 24 convolutional layers, and two fully-connected layers. When I ported this architecture into TensorFlow, I noticed my GPU was struggling memory-wise: TensorFlow would give numerous performance warnings related to an insufficient memory pool, and sometimes crash from running out of RAM on the GPU. YOLOv2 only has 9 convolutional layers, and has a smaller memory footprint.

In this paper it is claimed that YOLOv2 is more accurate than its bigger brother on various VOC test sets. However, this page says v2 actually performs a worse than its predecessor in terms of accuracy, because it has fewer layers. These contrasting claims made me wonder if more layers is really better in terms of accuracy.

Turning this question inside-out, one could ask if a deep, memory intensive architecture is really necessary if the problem is relatively simple (like detecting only a single object class, instead of one thousand), while maintaining good accuracy. Intuitively I'd say that the detection of objects defined by fewer features requires fewer layers, but when it comes to deep learning, intuition has failed me on several occasions already.

At present, I have yet to start training with either architecture, so I cannot directly perform experiments on my own input data and compare the outcomes. I had hoped there would be a general rule of thumb or general guidelines for a situation like this. The comments below seem to suggest there aren't. But perhaps after this clarification, someone could offer some more advice.

Answer 1

The only way to know is to try it and see. There is no general theory that allows us to reliably predict the outcome of such experiments.

That said, I question the motivation. You say that it feels "a bit heavy", but it's not entirely clear to me what that concretely means or why it would matter. Perhaps it is worth mentioning that even with dozens of layers, neural networks can be surprisingly fast. Remember the warning about premature optimization: don't let a general fear or assumption that the simple solution might be slow tie you up in knots and lead you to overly complex solutions. Instead, in this case I suggest you try the simplest solution first, and measure how well it performs. Only if you have evidence that it fails in some concrete, measurable way would you then consider more complex approaches. This will help you avoid unnecessarily complicating your life.

Finally, remember that there are other approaches to computer vision other than deep neural networks. People were doing object detection before neural networks became popular, and there are other methods that might be effective for detecting simple geometric shapes (e.g., morphological operators, Hough transform, shape descriptors, and more). But try the simple things first, and measure.

Answer 2

A general rule of thumb would be: look at the most successful architectures for your given task, take it as a prior and try to vary (by retraining) the number of layers if you need a specific point in the accuracy-speed tradeoff curve.

Previously people believed that one can get more accuracy just by stacking more layers. Even though this is true in most cases, however one can also vary the number of neurons/filters as was done in Wide Resnets. In that paper the authors increased the width (filters) of the Resnet (which is one of the most widely used backbone architectures) and could get the same accuracy results with wide and shallow networks (20-50 layers) as much deeper networks (100+ layers).