9
$\begingroup$

Why are neural networks initial weights initialized as random numbers? I had read somewhere that this is done to "break the symmetry" and this makes the neural network learn faster. How does breaking the symmetry make it learn faster?

Would'nt initializing the weights to 0 be a better idea? That way the weights would be able to find their values (whether positive or negative) faster?

Is there some other underlying philosophy behind randomizing the weights apart from hoping that they would be near their optimum values when initialized?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

The basic intuition behind initializing weight layers into small (and different) values is just so that the bias of the system is broken and weight values can move along and away and apart to different values.

More concretely, you'ld probably want your initial weights to be distinct and have "a small gap" between them, this 'gap' expands out as you go along and forces the weights to be a bit larger at every iteration, and this helps the network to converge faster, i.e. the learning process speeds up.

If you would instead have all your weights to some constant, each weight will be updated at a very slow (~fixed) rate, and this won't help much, specially if the initial values are 'very far' from the final values.

Hope that helps, Have fun learning :)

Improve this answer
$\endgroup$
4
  • $\begingroup$ So what you're saying is randomizing the initial weights is equivalent to giving each weight a nudge in the direction it needs to move ( and the gap to expand). $\endgroup$
    – Shayan RC
    Aug 23, 2013 at 7:11
  • $\begingroup$ I don't think it needs to be in the correct direction, you may as well start with a init weight of [-0.5 , +0.5] where the final values may be [+0.5, -0.5], the key idea is having different values.. $\endgroup$
    – Subhayan
    Aug 23, 2013 at 11:09
  • $\begingroup$ It's been over an year since I worked with NNs, so I am talking in a hand wavy manner, please let me know if you want the math behind it. but I think it is more important to get the intuition here, the math is pretty much available everywhere.. :) $\endgroup$
    – Subhayan
    Aug 23, 2013 at 11:11
  • $\begingroup$ The math behind it would be helpful but even more helpful would be some practical advice: Like how small should the initial weights be (10^?) How do they vary for different kind of networks? Is there some kind of magic numbers which work for all? $\endgroup$
    – Shayan RC
    Aug 23, 2013 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.