My question is a bit on the philosophical side, and there is probably not one single 'correct' answer on this. Nonetheless, I'm curious to hear your opinion...
I'm currently designing a convolutional neural network to locate features in images taken by a microscope. These features are geometrically quite simple and similar to one another, with only minor contextual variations from image to image. When I look at modern general-purpose covnets, they often have tens of convolution/pooling layers, making them a bit heavy. I can imagine that one needs many layers of abstraction when detecting cars, cats, and cods all in once, which are very distinct from one another.
Returning now to my more specialist network: would it need as many conv/pool layers as a general-purpose covnet? Or should I consider a different strategy all together (like having more fully-connected nodes)?
EDIT: Based on the comments, I feel I should rephrase/clarify my question a bit. Recently I looked into the YOLO architecture, which seems to strike a nice balance between accuracy and speed. The original version of YOLO features 24 convolutional layers, and two fully-connected layers. When I ported this architecture into TensorFlow, I noticed my GPU was struggling memory-wise: TensorFlow would give numerous performance warnings related to an insufficient memory pool, and sometimes crash from running out of RAM on the GPU. YOLOv2 only has 9 convolutional layers, and has a smaller memory footprint.
In this paper it is claimed that YOLOv2 is more accurate than its bigger brother on various VOC test sets. However, this page says v2 actually performs a worse than its predecessor in terms of accuracy, because it has fewer layers. These contrasting claims made me wonder if more layers is really better in terms of accuracy.
Turning this question inside-out, one could ask if a deep, memory intensive architecture is really necessary if the problem is relatively simple (like detecting only a single object class, instead of one thousand), while maintaining good accuracy. Intuitively I'd say that the detection of objects defined by fewer features requires fewer layers, but when it comes to deep learning, intuition has failed me on several occasions already.
At present, I have yet to start training with either architecture, so I cannot directly perform experiments on my own input data and compare the outcomes. I had hoped there would be a general rule of thumb or general guidelines for a situation like this. The comments below seem to suggest there aren't. But perhaps after this clarification, someone could offer some more advice.