This article says the following:

Deciding between the sigmoid or tanh will depend on your requirement of gradient strength.

I have seen (so far in my learning) 7 activation functions/curves. Each one seems to be building on the last. But then like the quote above, I have read in many places essentially that "based on your requirements, select your activation function and tune it to your specific use case".

This doesn't seem scalable. From an engineering perspective, a human has to come in and tinker around with each neural network to find the right or optimal activation function, which seems like it would take a lot of time and effort. I've seen papers which seem to describe people working on automatically finding the "best" activation function for a particular data set too. From an abstraction standpoint, it's like writing code to handle each user individually on a website, independently of the others, rather than just writing one user authentication system that works for everyone (as an analogy).

What all these are papers/articles are missing is an explanation of why. Why can't you just have one activation function that works in all cases optimally? This would make it so engineers don't have to tinker with each new dataset and neural network, they just create one generalized neural network and it works well for all the common tasks today's and tomorrow's neural networks are applied to. If someone finds a more optimal one, then that would be beneficial, but until the next optimal one is found, why can't you just use one neural network activation function for all situations? I am missing this key piece of information from my current readings.

What are some examples of why it's not possible to have a keystone activation function?


1 Answer 1


Those are old articles. Tinkering with activation functions probably isn't your best use of time, in most cases. Today, the standard engineering practice is (to a first order of approximation): use ReLU and don't stress over it. ReLU is clearly superior to sigmoid and tanh for most cases, so if you read older articles they'll talk about sigmoid and tanh, but today, ReLU has replaced them. There are fancier newer activation functions that in some cases are slightly better than ReLU and in some cases are slightly worse but the short version is ReLU is good enough and don't worry about the others at this stage in your learning and knowledge; just use ReLU and call it a day.

This is a crude simplification and there are absolutely exceptions but I'm giving you a rule of thumb that will be pretty reasonable in practice.

Why? My main answer is that you will need to get used to the fact that when working with neural networks, we don't really know the answer to most "why" questions. Sometimes we have intuition and theories but at its heart this is an empirical science: we don't really understand why neural networks work well. There are papers that give some explanation of why ReLU seems to do better than sigmoid/tanh -- in particular, sigmoid/tanh suffer from vanishing gradients when their inputs are in the tails of the sigmoid/tanh (as then their output is exponentially small, so the gradient is essentially zero), and then training gets stuck or proceeds very slowly -- but don't expect great theory that is going to tell you what to do. Instead, this is largely an empirical science, and if we're lucky, we have experiments and theory that helps us understand the empirical data we see.

I don't see any reason to expect there to be a single activation function that is optimal for all tasks, so I'm not bothered if that isn't true and don't feel that we need a "reason" for it to be false.

  • $\begingroup$ So you're saying that ReLU should be used and is (for all intents and purposes) the most optimal activation function for all types of tasks? Which means you're saying that you don't need to touch it for different neural networks, so there basically is a keystone one (for now, the ReLU)? Also, thank you for the practical advice, that will help save a ton of time :) $\endgroup$
    – Lance
    Aug 19, 2020 at 21:12
  • $\begingroup$ "better than ReLU and in some cases are slightly worse" this is partly what I'm wondering, is there or isn't there something that works in all cases? $\endgroup$
    – Lance
    Aug 19, 2020 at 21:17
  • 2
    $\begingroup$ @LancePollard, no, I'm not saying it is necessarily optimal. Nothing is optimal in all cases but there is nothing that is clearly superior to ReLU, and from an engineering sense "always use ReLU" is a simple and common and reasonable strategy for now. $\endgroup$
    – D.W.
    Aug 20, 2020 at 0:18
  • 1
    $\begingroup$ Isn't ReLU also susceptible to vanishing gradient? (On the negative side) I would have thought LeakyReLU with a small slope on the negative side must perform just as well without having this issue. $\endgroup$
    – Tassle
    Aug 20, 2020 at 13:37
  • 1
    $\begingroup$ @Tassle, yup, I'm not sure that we fully understand why ReLU performs so well, and I think your puzzlement and your intuition is reasonable and appropriate. Like I said, we often don't have explanations to fully explain what we observe empirically. I suggest posting a separate question if you'd like to get into those details. Nonetheless, I stand by statement that ReLU is basically about as good as anything else, in most cases, even if we don't know exactly why, and I stick with my advice to just use ReLU and not worry about it when you're first starting out in this field remains $\endgroup$
    – D.W.
    Aug 20, 2020 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.