2
$\begingroup$

In the Deep MNIST Tutorial for Tensor Flow, (and in general), we create a convolutional layer with a weight tensor, say W, of shape (patch dims, #input channels, # output channels). However, when we define the bias tensor, b, its shape is only (#output channels).

When we convolve W and the input x, with correct padding, we will get a volume with depth = #output channels. How can we add a single bias vector to such a volume?

Are we supposed to expand the bias vector to take on the spatial dimensions of the input x?

$\endgroup$
  • $\begingroup$ What kind of answer are you looking for? Are you looking for code that will work in some specific library for deep learning? (Code is off-topic here.) Are you looking for an equation? Why are you finding it difficult to add a bias vector? Are you basically are asking "how do I add two d-dimensional vectors"? $\endgroup$ – D.W. Jun 18 '16 at 2:11
1
$\begingroup$

No. The bias vector should indeed have the same dimensions as the output vector (not the dimensions of the input). In a neural network, after summing the weights times the inputs, the bias is added pointwise, then you apply the activation function (e.g., ReLu). The equation for a single neuron is something like

$$y = f(\sum_i c_i x_i + c_b)$$

where $x_1,\dots,x_n$ are the inputs, $c_1,\dots,c_n$ are the weights on the inputs, $c_b$ is the bias, and $f$ is the activation function. Notice how we have one bias number per neuron, i.e., per output. (This equation gets modified a bit for convolutional neural networks but the relevant part still remains the same: we have one bias value per output.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.