0
$\begingroup$

I want to understand how the backpropagation algorithm would work on a neural network with multiple outputs.

More specifically, I have a network with 21 binary (0/1) outputs and I want to minimize the number of outputs that I get correctly; in other words, I want to minimize the hamming distance between the output vector and the desired vector.

How does the loss function work here? How do I backpropagate the error and update the weights? I know this might be long to explain so I'm also happy with links to some good references that I could read on the matter.

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ As far as I know, most real-world implementations use a continous output range $[0, 1]$ instead of the discrete set $\{0,1\}$. This allows to define a continuous cost function whose value can then be minimized. (The hamming distance is not continuous.) $\endgroup$ – still_learning Feb 26 '17 at 18:01
  • $\begingroup$ You can use softmax for multiple output problem. $\endgroup$ – iLoveCamelCase Mar 8 '17 at 17:28
1
$\begingroup$

You have to pick a loss function before you can apply backpropagation or train your network. Once you do, you can apply backpropagation. Backpropagation doesn't tell you how to pick a loss function; that's something you have to do, based on what you're trying to achieve.

Neural networks don't have binary outputs. Rather, they have continuous outputs. You might want to use a logistic loss on each of the 21 outputs, and sum up those 21 losses and use that as your overall loss function. The logistic loss is effectively a generalization of the 0-or-1 loss to the case where you have a continuous output and you want to predict either 0 or 1.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.