I'm programming a neural network. I know that I should initialize the network by picking random weights. How do I pick a random weight for the connections to bias nodes? What distribution should I use for these weights? I can pick a random value from the range $[0,U]$, but what value should I use for the upper limit $U$?

What I've tried: I've set $U$ to correspond to the number of inputs to a neuron. Then the percentage of inputs '$K$' that the node needs to fire (this is set in the agents genome) which is set from 0.0 to 1.0, is multiplied with $U$ to get the weight of the bias node. So a node with a $K$ value of 0.7 and 40 inputs would have a bias weight of $U * K = 0.7*40 = 28$. Am I correct for assuming that $U$ would be the total number of a nodes inputs? Or am I approaching this wrong?

  • $\begingroup$ Either I don't understand your question or the terminology is mixed up. From my experience, a bias node is an extra node on a layer. The node's value is always 1.0 (this doesn't change). When the network is being trained, the weight of that node serves to nudge the input values of the next layer's nodes. $\endgroup$ – ZeroUltimax Jul 2 '15 at 0:38
  • $\begingroup$ The value of the node is indeed 1. I was talking about the weight of the connection. When I'm randomly setting values for the weight of the bias nodes, how do I know what the upper limit would be. I can't just set the weight to something like random(100) $\endgroup$ – Samuel Mungy Jul 2 '15 at 8:19
  • $\begingroup$ Plus, how do people set the lower limits and upper limits of their weights as well? How do you decide on that? $\endgroup$ – Samuel Mungy Jul 3 '15 at 15:53
  • $\begingroup$ This question is confusing. What do you mean by 'agents genome'? I'm not familiar with that phrase used in the context of neural networks. When you talk about 'agents genome', that makes me wonder if you are trying to do some kind of genetic programming. Are you trying to do some combination of neural networks and genetic programming? Normally a node in a neural network doesn't have a percentage of inputs it needs to figure; it has an activation function (e.g., sigmoid, etc.), so I'm not sure what you mean by that, either. Are we missing some context? $\endgroup$ – D.W. Jul 3 '15 at 19:46
  • $\begingroup$ I'm really sorry. Yes it's a mix with genetic programming. The question is just simply asking, how do people know what they would use for their upper bound and lower bound of their weights. $\endgroup$ – Samuel Mungy Jul 4 '15 at 7:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.