How is ReLU used in machine learning functions

Question

I've seen how sigmoid would be used in machine learning

sigmoid(dot(activations, weights)-bias)

like this ^

but sigmoid makes sure your values are between 0 and 1. So how would ReLU ( max(0,x) ) be swapped out for sigmoid, if it doesn't clamp your values to a 0 to 1 range?

Or would I just use tanh(x) in the output layer?

My initial guess is that it would just look like this:

ReLU(dot(activations, weights)-bias) // all values are from 0 to +inf

Lastly, what is the traditional range for node biases? I'm guessing -1 to 1 just like weights.

Answer 1

ReLU is used in all layers except at the very end. Normally softmax is used at the final output, to normalize the outputs to be in the range [0,1] and to ensure the outputs sum to 1.