Applying a 1x1 convolution on an existing layer

Question

I am looking at the following github. In this, layer3 has a shape of (1, 20, 72, 256) and I interpret this is as a single layer having 20 input nodes and mapping to 72 output nodes and there are $256$ filters, so the depth is $256$. Then, after applying a 1x1 convolution to this, the new shape becomes (1, 20, 72, 2). I don't quite understand how this comes about.

Is it because there are only 2 classes in the model, so it only needs a depth of two at all points? I keep stumbling over the code and can't get anywhere.

Answer 1

A $1 \times 1$ convolution is a modern technique for dimensionality reduction, originally proposed in the Inception paper by Google team.

Suppose the layer outputs a $(N,H,W,F)$ tensor, where $F$ is the number of feature maps. Then the number of floating-point operations that the subsequent convolutional layer is going to compute is $O(N \cdot H \cdot W \cdot F)$, most importantly it's proportional to $F$. Likewise the number of parameters is also proportional to $F$ and the filter size.

The problem is that deep convolutional networks tend to increase the number filters exponentially, so $F$ can be really large, which makes training deep networks pretty expensive.

The idea is to reduce the size of the tensor via clever application of the convolution to $(N,H,W,F')$, where $F'$ is much smaller than $F$. If the network learns to compress the information of $F$ feature maps into $F'$ feature maps, the whole network will become much lighter both in terms of memory and computational complexity, yet potentially deeper. As it turns out, this trick works magically, so that it's used all over the place before expensive convolutional layers:

One big problem with the above modules, at least in this naive form, is that even a modest number of 5x5 convolutions can be prohibitively expensive on top of a convolutional layer with a large number of filters.

This leads to the second idea of the proposed architecture: judiciously applying dimension reductions and projections wherever the computational requirements would increase too much otherwise. This is based on the success of embeddings: even low dimensional embeddings might contain a lot of information about a relatively large image patch...1x1 convolutions are used to compute reductions before the expensive 3x3 and 5x5 convolutions. Besides being used as reductions, they also include the use of rectified linear activation which makes them dual-purpose.

The trick itself is pretty simple: apply $F'$ convolutions with $1 \times 1$ kernel size. This doesn't change spatial dimensions of the image, but only reduces the depth.

This is exactly what happened in your case: the depth was reduced from 256 to 2, which is a 128x boost for the subsequent layer.