I have been recently thinking about activation functions and the explainability.

For sigmoid and tanh activation functions, I am thinking of them to be similar to logistic regression as the output of the activation function is very close to binary. So, to me the neural network is making simultaneous decisions and that we are training the neural network to be making better decisions.

But then the meaning behind Relu and related activation functions are lost on me. I don't quite get what the motivation behind them other than the fact that they have faster training times. So can someone enlighten some deeper meaning behind the activation functions?

  • $\begingroup$ Relu is close to the positive part of the signal, while being smooth. It discards negative inputs. A faster training time is not the purpose. Note that activation function is a generic term, valid for any function. $\endgroup$
    – user16034
    Jun 7, 2023 at 8:42
  • $\begingroup$ @Yves Daoust, I understand what the function does, but is there any meaning behind only taking the positive? as with my original post, my reasoning is that the sigmoid and tanh activation functions can be interpreted as making the values binary which can be thought of as making decisions, accepting or rejecting a decision. but I don't see any equivalent meaning behind relu, I feel like I might be missing something very big but don't know what $\endgroup$ Jun 7, 2023 at 9:25
  • $\begingroup$ Relu is also a decision: keep positive, reject negative You can also see it as the signal times a binary decision. But in fact, any smooth non-linear function can be used, with no need for a clear interpretation. $\endgroup$
    – user16034
    Jun 7, 2023 at 9:38
  • $\begingroup$ Yes, I know any differentiable function can be used, but then the function would lose interpretability so I m just thinking, something might produce a good metric evaluation but it loses interpretability and so loses "correctness". Because to me, it seems like they are losing logical reasoning in the calculations. Like choosing positive is the same as partitioning a high dimensional space into two, but the split is kind of arbitrary and thus the different versions of RELU? $\endgroup$ Jun 7, 2023 at 11:06
  • $\begingroup$ A sigmoid is also partitioning space in two. $\endgroup$
    – user16034
    Jun 7, 2023 at 12:02

1 Answer 1


There is no deeper meaning from an interpretability perspective. Neural networks are, generally speaking, not interpretable. They are universal function approximators, which means they are a way to approximate any function. You don't have to care about how they approximate it, and you shouldn't expect any explanation of how they approximate it or what function they are approximating.

The final layer is special and plays a special role. The softmax function has a particular interpretation. So too does the sigmoid function, which is a version of the softmax for binary classification. You should not expect a similar explanation for activation functions at intermediate layers, because the final layer / final output is unique.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.