# meaning behind activation function

I have been recently thinking about activation functions and the explainability.

For sigmoid and tanh activation functions, I am thinking of them to be similar to logistic regression as the output of the activation function is very close to binary. So, to me the neural network is making simultaneous decisions and that we are training the neural network to be making better decisions.

But then the meaning behind Relu and related activation functions are lost on me. I don't quite get what the motivation behind them other than the fact that they have faster training times. So can someone enlighten some deeper meaning behind the activation functions?

• Relu is close to the positive part of the signal, while being smooth. It discards negative inputs. A faster training time is not the purpose. Note that activation function is a generic term, valid for any function.
– user16034
Jun 7, 2023 at 8:42
• @Yves Daoust, I understand what the function does, but is there any meaning behind only taking the positive? as with my original post, my reasoning is that the sigmoid and tanh activation functions can be interpreted as making the values binary which can be thought of as making decisions, accepting or rejecting a decision. but I don't see any equivalent meaning behind relu, I feel like I might be missing something very big but don't know what Jun 7, 2023 at 9:25
• Relu is also a decision: keep positive, reject negative You can also see it as the signal times a binary decision. But in fact, any smooth non-linear function can be used, with no need for a clear interpretation.
– user16034
Jun 7, 2023 at 9:38
• Yes, I know any differentiable function can be used, but then the function would lose interpretability so I m just thinking, something might produce a good metric evaluation but it loses interpretability and so loses "correctness". Because to me, it seems like they are losing logical reasoning in the calculations. Like choosing positive is the same as partitioning a high dimensional space into two, but the split is kind of arbitrary and thus the different versions of RELU? Jun 7, 2023 at 11:06
• A sigmoid is also partitioning space in two.
– user16034
Jun 7, 2023 at 12:02