What determines the number of inputs and outputs when initialising weights in a convolutional neural network?

Question

Following Deep MNIST for Experts tutorial on Tensorflow, I realize I don't understand where the choice of numbers comes from when initializing weights.

In the tutorial, they first show the below function for creating random weights.

def weight_variable(shape):
  initial = tf.truncated_normal(shape, stddev=0.1)
  return tf.Variable(initial)

However, the shape parameter has me confused when the tutorial shows the next line of code:

W_conv1 = weight_variable([5, 5, 1, 32])

The tutorial says the following:

The first two dimensions are the patch size, the next is the number of input channels, and the last is the number of output channels.

I don't understand the significance of the last two values, 1 and 32.

What decides what values you input there? And what do they affect?

When I'm designing my own neural network, could I put any value I like for the last two, or if I were to change them in a neural network would it 'break' the entire network?

Answer 1

The question is about basic principles of Convolutional Neural Networks, specifically how the layers are organized (it's best explained here, in fact I highly recommend to complete the whole CS231n course, it's amazing!).

What decides what values you input there? and what do they affect?

W_conv1 = weight_variable([5, 5, 1, 32])

The input channel of the first convolutional layer is the depth of MNIST images. Since they are grayscale, there's only one channel. If you had colorful CIFAR-10 images, you'd use 3 channels (R, G, B).

The 5x5 filters in this layer (32) would be the the output of this layer and input to the next layer. Indeed, as you can see the next layer defines

W_conv2 = weight_variable([5, 5, 32, 64])

When I'm designing my own neural network, could I put any value I like for the last two, or if I were to change them in a neural network would it 'break' the entire network?

Yes, you have freedom here, you only need to make input-output channels consistent. If you decide to use 48 filters instead 32, you would write:

W_conv1 = weight_variable([5, 5, 1, 48])
b_conv1 = bias_variable([48])
...
W_conv2 = weight_variable([5, 5, 48, 64])
b_conv2 = bias_variable([64])

Like I said before, the first 1 is determined by your data. But you can change the filter size as well, from 5x5 to 3x3.

Note that if the number of channels doesn't match, the graph won't simply compile. You'll see an error even before the training starts, so don't be afraid to experiment.