Neural networks use specific activation functions, commonly used ones are tanh, ReLu.

I have seen that people have experimented with continuously parametrices activation functions, for instance here. Here, the additional parameter $\alpha$ which interpolates from logarithmus to linear to exponental, is optimized during the back-propagation step of the training.

I wonder whether much more general types of activation functions have been considered, such as Taylor-expansion of an unknown function where the coefficients are parameters in the training? It seems is a lot of degrees of freedom, that are not considered conventionally.

So my questions:

  1. What is known about optimimality of actionation functions, apart from empirally testing? Are there mathematical proofs, for instance, that show some necessary properties for ideal learning (for instance, is monotonicity necessary, or could a sinusoidal activation function lead to optimal results too?)

  2. If the answer to question 1 is no: Is there much research investigating the topic of learning activation functions?


1 Answer 1


Monotonicity is necessary! A neuron (perceptron) is either activated or not activated given an input. If you're trying to learn "red" and red should activate the neuron; the more red it's the most likely it should be activated (This monotonic gradient is what this activation function represents). If at certain redness it activates but at more redness it deactivates it makes no sense e.g. it can't learn red (The case for non-monotonic functions).

That they're differentiable functions makes it practical for the back propagation algorithm.

What they explain in your linked article is that they're trying to better approximate other functions e.g. sinusoidal. They're not trying sinuosoidal as an activation function. They're combining several monotonic functions to create yet another monotonic function, that is also differentiable.

  • 1
    $\begingroup$ Could you please refere to a source that shows that Monotonicity is necessary? I am not aware of any, and it is not clear to me why it should always be the case. In particular, there are examples where non-monocromatic functions have been used. You just provide an example where intuitively it seems that monotonicity is necessary. I would like to know about a proof that this is always the case. $\endgroup$ Feb 13, 2019 at 20:57
  • $\begingroup$ @NicoDean I'm afraid my answer was, as you said, based on intuition and some practical experience. I wasn't aware of the example you provide. It seems like an interesting possibility for time series functions. They do acknowledge though that they don't perform better than RNNs which use monotonic activation functions. $\endgroup$ Feb 14, 2019 at 13:06
  • 2
    $\begingroup$ I doubt there is any result that monotonicity is "necessary" in any strict sense. Rather, I think this is based on your intuition that you expect monotonic activation functions to perform better than non-monotonic ones. That kind of intuition doesn't seem unreasonable to me. But ultimately, these kinds of questions are empirical ones that can ultimately only be answered through experiments. $\endgroup$
    – D.W.
    Mar 15, 2019 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.