1
$\begingroup$

Say you have a neural net that is being trained using back propagation and you are using relu activation. Say the input to a node is a weighted sum of the previous layer with a bias term and say for a particular data point, this weighted sum plus bias is negative. Then relu returns 0. Notice the change in the loss as a function of the change in one of these weights or the bias is 0. Therefore the network won't improve the bias as the network does back propagation. Why is this not a problem?

$\endgroup$
1
$\begingroup$

Normally, any ReLU unit will return 0 for some samples and not for other samples (in particular, it'll probably return 0 for about half of the samples). Because of how SGD works, it won't get stuck: the former samples will yield a contribution to the gradient of 0, but the latter samples will have a non-zero gradient, so the weights and bias will be updated for the latter samples. So it's not a problem; the weights and bias still get updated for many of the samples.

That said, there can be cases where this problem can occur. If the ReLU unit outputs 0 for all (or most) samples, then SGD won't update the weights, and learning for that neuron will get stuck. This is sometimes called the "dying ReLU" problem. See, e.g., https://datascience.stackexchange.com/q/5706/8560 and https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Potential_problems. Typically we're able to avoid this problem by appropriate initialization of the weights and by suitable choice of learning rate for SGD, so it's usually not a big deal in practice if you choose those appropriately, but it can happen.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

"Typically we're able to avoid this problem by appropriate initialization of the weights and by suitable choice of the learning rate for SGD"

As far as I know, this approach is slightly outdated and is something we are trying to move from.

In various cases, it is recommended to use LeakyRelu, which aims to deal exactly with the issue you mentioned.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Do you have any evidence you can share to support your statement that we are trying to move away from this approach, or that there is any community consensus for your claim about LeakyReLU? $\endgroup$ – D.W. Jun 26 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.