Why isn't there just one "keystone" activation function in Neural Networks?

Question

This article says the following:

Deciding between the sigmoid or tanh will depend on your requirement of gradient strength.

I have seen (so far in my learning) 7 activation functions/curves. Each one seems to be building on the last. But then like the quote above, I have read in many places essentially that "based on your requirements, select your activation function and tune it to your specific use case".

This doesn't seem scalable. From an engineering perspective, a human has to come in and tinker around with each neural network to find the right or optimal activation function, which seems like it would take a lot of time and effort. I've seen papers which seem to describe people working on automatically finding the "best" activation function for a particular data set too. From an abstraction standpoint, it's like writing code to handle each user individually on a website, independently of the others, rather than just writing one user authentication system that works for everyone (as an analogy).

What all these are papers/articles are missing is an explanation of why. Why can't you just have one activation function that works in all cases optimally? This would make it so engineers don't have to tinker with each new dataset and neural network, they just create one generalized neural network and it works well for all the common tasks today's and tomorrow's neural networks are applied to. If someone finds a more optimal one, then that would be beneficial, but until the next optimal one is found, why can't you just use one neural network activation function for all situations? I am missing this key piece of information from my current readings.

What are some examples of why it's not possible to have a keystone activation function?

Answer 1

Those are old articles. Tinkering with activation functions probably isn't your best use of time, in most cases. Today, the standard engineering practice is (to a first order of approximation): use ReLU and don't stress over it. ReLU is clearly superior to sigmoid and tanh for most cases, so if you read older articles they'll talk about sigmoid and tanh, but today, ReLU has replaced them. There are fancier newer activation functions that in some cases are slightly better than ReLU and in some cases are slightly worse but the short version is ReLU is good enough and don't worry about the others at this stage in your learning and knowledge; just use ReLU and call it a day.

This is a crude simplification and there are absolutely exceptions but I'm giving you a rule of thumb that will be pretty reasonable in practice.

Why? My main answer is that you will need to get used to the fact that when working with neural networks, we don't really know the answer to most "why" questions. Sometimes we have intuition and theories but at its heart this is an empirical science: we don't really understand why neural networks work well. There are papers that give some explanation of why ReLU seems to do better than sigmoid/tanh -- in particular, sigmoid/tanh suffer from vanishing gradients when their inputs are in the tails of the sigmoid/tanh (as then their output is exponentially small, so the gradient is essentially zero), and then training gets stuck or proceeds very slowly -- but don't expect great theory that is going to tell you what to do. Instead, this is largely an empirical science, and if we're lucky, we have experiments and theory that helps us understand the empirical data we see.

I don't see any reason to expect there to be a single activation function that is optimal for all tasks, so I'm not bothered if that isn't true and don't feel that we need a "reason" for it to be false.