I am trying to understand the concepts behind the InceptionNet V3 and got confused with the meaning of representational bottleneck. They said

One should avoid bottlenecks with extreme compression. In general the representation size should gently decrease from the inputs to the outputs before reaching the final representation used for the task at hand.

and one thing I get from this is that we should not use polling or convolutions with large strides. But when I arrived in the section 5 (Efficient Grid Size reduction) they said that doing polling before the convolution violates that mentioned principle but doing polling after convolution don't violate.

Doing the left one we decrease the number of features by a factor of one quarter and then the features are doubled. In other words a decrease of 35*35*320*0.75 = 294K features and then an increase of 17*17*320 = 92.48K features. And doing the one in the right the features are doubled and then decrease by the same factor of one quarter. And doing this one, the real decrease in the pooling phase is bigger: 35*35*640*0.75 = 588K.

So, why one approach violates the principle and the other don't? What am I missing here?

Another thing that confused me is the solution found by them to avoid the computational cost of doing conv. before polling.

The polling part of the solution is not violating the principle of representational bottleneck once it make the exactly same operation with dimensionality reduction mentioned earlier?

  • $\begingroup$ Please ask only one question per post. $\endgroup$ – D.W. Jul 26 '18 at 19:17
  • $\begingroup$ According to this post I think the questions are about the same issue (violation of representational bottleneck) $\endgroup$ – Hemerson Tacon Jul 26 '18 at 20:32

Cool question! I managed to start thinking about its meaning thanks to your asking.

I believe you would like to consider it in the sense of information flow. As the information is compressed to a same size in both cases, the question is which one is more effective. Information with amount of 35*35*640 can flow through a wider pipe more freely. A pooling layer just prunes information directly in the left solution, while doing Inception in the right case expands the representation to a more sparse space and makes meaningful inferences before get pruned. Think of the pooling operation in left as a neck or a narrow path, and the middle representation in right as a broad lake. The pooling layer in right can also select better output since the source has more abstract (semantics) information.

As to figure 10, I consider it as a balance between the representation size and the computation burden.

  • $\begingroup$ Your first answer clarified a bit more the purpose of the right side alternative to me. But about the second, I still not see how it could guarantee that the information would be semantically different in those 2 branches and then keep avoiding representational bottleneck. $\endgroup$ – Hemerson Tacon Jul 26 '18 at 15:05

The usual rule is one question per post. I'll answer your first question.

With the architecture shown on the left, we first push everything down to 17x17x320, losing a lot of information. Then we expand to 17x17x640, but once information is gone, it can't be reconstructed, so with the left architecture, effectively the narrowest pipe is 17x17x320.

With the architecture shown on the left, we never go down to 17x17x320; the narrowest part of the pipe is 17x17x640. 17x17x640 potentially allows representing twice as much information as 17x17x320. Therefore, with the right architecture, we haven't lost as much information as with the left architecture.

In particular, the left architecture probably loses about twice as much information as the right architecture. It's bottleneck is twice as "narrow". So, it might have lower accuracy as a result.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.