1
$\begingroup$

this may seem to be a pretty basic question, but it is something i have been puzzling over for some time.

when calculating the activations of nodes in a hidden layer in an ANN using sigmoid neurons for use with the backpropagation algorithm - should the output of the neuron be thresholded or not?

say the activation vector for layer $i$ is calculated by:

$\mathbf{a}_i = \sigma(\mathbf{w}_i\mathbf{a}_{i-1} + \mathbf{b}_i)$

where $\sigma$ is the sigmoid activation function

should it actually be:

$\mathbf{a}_i = (\sigma(\mathbf{w}_i\mathbf{a}_{i-1} + \mathbf{b}_i) > 0.5$)

to ensure that $\mathbf{a}_i$ is binary vector? or should the real value from $\sigma$ be passed on to the next layer?

my main reason for asking this is that biological neurons don't transmit real valued data, but i haven't really been able to find any definitive answer anywhere to this question or anyone who explicitly says to threshold the value; it seems like a pretty fundamental question to the functioning of ANNs, so either i am missing something in my reading, or it is considered so common-sense that it doesn't need to be mentioned much.

any help would be greatly appreciated

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

When you do thresholding your activation function becomes a step function. The problem with that is that step functions are not differentiable, so you would not be able to do basic gradient descent to learn the parameters. You can see the sigmoid as a continuous approximation of a step function.

Improve this answer
$\endgroup$
2
  • $\begingroup$ okay, thank you for responding - so the outputs of the neurons should be not converted to binary then, i suppose i was getting too caught up on the analogue of biological neurons. should i use a scalar multiplier to the input of the sigmoid function to make the function more step-like? or is it best to just leave it as is? $\endgroup$
    – guskenny83
    Oct 8, 2014 at 9:00
  • $\begingroup$ Usually, the learned parameters should reflect the scaling of inputs and hence you do not need to set this by hand. $\endgroup$
    – Tobias
    Oct 8, 2014 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.