8
$\begingroup$

A lot of activation functions in neural networks (sigmoid, tanh, softmax) are monotonic, continuous and differentiable (except of may be a couple of points, where derivative does not exist).

I understand the reason for continuity and differentiability, but can not really understand a reason for monotonisity.

$\endgroup$
8
$\begingroup$

During the training phase, backpropagation informs each neuron how much it should influence each neuron in the next layer. If the activation function isn't monotonic then increasing the neuron's weight might cause it to have less influence, the opposite of what was intended. The result would be choatic behavior during training, with the network unlikely to converge to a state that yields an accurate classifier.

$\endgroup$
  • $\begingroup$ Just to clarify: Gradient descent finds a local minimum even with monotonic activation functions. It might only take longer. $\endgroup$ – Martin Thoma Dec 6 '15 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.