Back propagation in neural networks

Question

I just finished watching these 3 Coursera videos on back propagation in neural networks. I get the idea of what we're trying to do, but I don't get how we achieve that by calculating error in each step as weight * cascaded error (eg. the formula at the top right of screen at 12:07 in the linked video). Let's say we start off with all the weights (theta) at zero. Wouldn't back propagation always calculate 0 error for everything, causing nothing to change?

Answer 1

Weights must be initialized to random, distinct, small values.

If all the weights are initialized to zero, our optimization function (eg. gradient descent) will not work. According to a later video, it's a problem of symmetry that will happen when you initialize weights to the same value. So if you used 0.4 instead of zero, the same problem would happen.

From course material:

When training neural networks, it is important to randomly initialize the parameters for symmetry breaking. One effective strategy for random initialization is to randomly select values for weights uniformly in the range [-x, x]. One effective strategy for choosing x is to base it on the number of units in the network. A good choice of x is x = sqrt(6) / sqrt(in + out) where in and out are the number of units in the layers adjacent to [the weights]. This range of values ensures that the parameters are kept small and makes the learning more efficient.