How to determine the dimensions of the bias vector in a Convolutional layer (tensor flow)?

Question

In the Deep MNIST Tutorial for Tensor Flow, (and in general), we create a convolutional layer with a weight tensor, say W, of shape (patch dims, #input channels, # output channels). However, when we define the bias tensor, b, its shape is only (#output channels).

When we convolve W and the input x, with correct padding, we will get a volume with depth = #output channels. How can we add a single bias vector to such a volume?

Are we supposed to expand the bias vector to take on the spatial dimensions of the input x?

Answer 1

No. The bias vector should indeed have the same dimensions as the output vector (not the dimensions of the input). In a neural network, after summing the weights times the inputs, the bias is added pointwise, then you apply the activation function (e.g., ReLu). The equation for a single neuron is something like

$$y = f(\sum_i c_i x_i + c_b)$$

where $x_1,\dots,x_n$ are the inputs, $c_1,\dots,c_n$ are the weights on the inputs, $c_b$ is the bias, and $f$ is the activation function. Notice how we have one bias number per neuron, i.e., per output. (This equation gets modified a bit for convolutional neural networks but the relevant part still remains the same: we have one bias value per output.)