I just finished watching these 3 Coursera videos on back propagation in neural networks. I get the idea of what we're trying to do, but I don't get how we achieve that by calculating error in each step as weight * cascaded error (eg. the formula at the top right of screen at 12:07 in the linked video). Let's say we start off with all the weights (theta) at zero. Wouldn't back propagation always calculate 0 error for everything, causing nothing to change?

  • 1
    $\begingroup$ I suggest that you come up with the simplest example you can (of the smallest neural network possible), and draw that together with the current weights and a specific input and show us your calculations for why you think back propagation will calculate 0 error. Then add that to the question so we can help you better. It won't always calculate 0 error, but unless we can see your reasoning, it's hard for us to know where you went wrong or what your misunderstanding is. $\endgroup$
    – D.W.
    Aug 19 '16 at 15:57
  • $\begingroup$ Well, the formula is weight * error, so if weight is zero, then the formula will return 0 regardless of what the error at the next node was. $\endgroup$ Aug 19 '16 at 22:24

Weights must be initialized to random, distinct, small values.

If all the weights are initialized to zero, our optimization function (eg. gradient descent) will not work. According to a later video, it's a problem of symmetry that will happen when you initialize weights to the same value. So if you used 0.4 instead of zero, the same problem would happen.

From course material:

When training neural networks, it is important to randomly initialize the parameters for symmetry breaking. One effective strategy for random initialization is to randomly select values for weights uniformly in the range [-x, x]. One effective strategy for choosing x is to base it on the number of units in the network. A good choice of x is x = sqrt(6) / sqrt(in + out) where in and out are the number of units in the layers adjacent to [the weights]. This range of values ensures that the parameters are kept small and makes the learning more efficient.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.