How to determine the dimensions of the bias vector in a Convolutional layer (tensor flow)?

In the Deep MNIST Tutorial for Tensor Flow, (and in general), we create a convolutional layer with a weight tensor, say W, of shape (patch dims, #input channels, # output channels). However, when we define the bias tensor, b, its shape is only (#output channels).

When we convolve W and the input x, with correct padding, we will get a volume with depth = #output channels. How can we add a single bias vector to such a volume?

Are we supposed to expand the bias vector to take on the spatial dimensions of the input x?

• What kind of answer are you looking for? Are you looking for code that will work in some specific library for deep learning? (Code is off-topic here.) Are you looking for an equation? Why are you finding it difficult to add a bias vector? Are you basically are asking "how do I add two d-dimensional vectors"?
– D.W.
Jun 18, 2016 at 2:11

$$y = f(\sum_i c_i x_i + c_b)$$
where $x_1,\dots,x_n$ are the inputs, $c_1,\dots,c_n$ are the weights on the inputs, $c_b$ is the bias, and $f$ is the activation function. Notice how we have one bias number per neuron, i.e., per output. (This equation gets modified a bit for convolutional neural networks but the relevant part still remains the same: we have one bias value per output.)