I've seen how sigmoid would be used in machine learning
sigmoid(dot(activations, weights)-bias)
like this ^
but sigmoid makes sure your values are between 0 and 1. So how would ReLU ( max(0,x) ) be swapped out for sigmoid, if it doesn't clamp your values to a 0 to 1 range?
Or would I just use tanh(x) in the output layer?
My initial guess is that it would just look like this:
ReLU(dot(activations, weights)-bias) // all values are from 0 to +inf
Lastly, what is the traditional range for node biases? I'm guessing -1 to 1 just like weights.