-1
$\begingroup$

I've seen how sigmoid would be used in machine learning

sigmoid(dot(activations, weights)-bias)

like this ^

but sigmoid makes sure your values are between 0 and 1. So how would ReLU ( max(0,x) ) be swapped out for sigmoid, if it doesn't clamp your values to a 0 to 1 range?

Or would I just use tanh(x) in the output layer?

My initial guess is that it would just look like this:

ReLU(dot(activations, weights)-bias) // all values are from 0 to +inf

Lastly, what is the traditional range for node biases? I'm guessing -1 to 1 just like weights.

|cite|improve this question
$\endgroup$
1
$\begingroup$

ReLU is used in all layers except at the very end. Normally softmax is used at the final output, to normalize the outputs to be in the range [0,1] and to ensure the outputs sum to 1.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Also, is it normal if when I use ReLU, that the values of the nodes will often be 0, since the weights are between -1 and 1 $\endgroup$ – Andrew900460 Feb 14 '18 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.