I have been recently thinking about activation functions and the explainability.
For sigmoid and tanh activation functions, I am thinking of them to be similar to logistic regression as the output of the activation function is very close to binary. So, to me the neural network is making simultaneous decisions and that we are training the neural network to be making better decisions.
But then the meaning behind Relu and related activation functions are lost on me. I don't quite get what the motivation behind them other than the fact that they have faster training times. So can someone enlighten some deeper meaning behind the activation functions?