4
$\begingroup$

Can I utilize the backpropagation algorithm in a layered, feed-forward ANN in instances where there are multiple output neurons? If so, how? Links to (somewhat) comprehensible resources would be greatly appreciated, if nothing else.

The background is that I'm working on a simple C program that can create, propagate, and train dynamically sized, layered feed-forward ANNs. For the most part, everything's been going swimmingly. However, I'm having a little trouble wrapping my head around backpropogation. My main concern right now is how to use the backpropagation method for training a network that has multiple output neurons. All the examples/explanations I've found only use one output neuron. I'm assuming this is for the sake of simplicity. But, I'm wondering if, perhaps, this is because the backpropagation algorithm is only designed to works with one output neuron at a time. In other words, it can only be applied relative to one output neuron every time an feed-forward ANN is propagated. This would make sense, as it is known to be among the simplest ANN training methods.

Here are links to my aforesaid resources (or at least, the ones I found intuitive):

Thanks!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

Backpropagation works for feed-forward ANNs with multiple output neurons.

For an output neuron $z_r$ in an ANN, the delta function is

$$ \delta_r = \frac{\partial f_r(a_r)}{\partial a_r} e_r\qquad, $$ where $f_r(a_r)$ is the activation function of $z_r$, $a_r$ is the sum of weighted inputs of $z_r$ and $e_r$ is the error of $z_r$, $e_r$ = $z_r$ - $y_r$ ($y_r$ is the target value).

for a hidden neuron $z_k$ the delta function is the sum of weighted deltas of the neurons in the next layer (in forward direction) of $z_k$. So, for a neuron in the layer next to the output layer, the delta function is:

$$ \delta_k = \frac{\partial f_k(a_k)}{\partial a_k} \sum_{r\in post(k)}\delta_r w_{kr}\qquad, $$

where $w_{kr}$ is the weight between neurons $z_k$ and $z_r$. I suggest to have a look at the Wikipedia article about Backpropagation, especially the subsection 'finding the derivative of the error'. Following the link at the end of this section (https://www4.rgu.ac.uk/files/chapter3%20-%20bp.pdf), the first five pages or so might be helpful for a better understanding.

Improve this answer
$\endgroup$
2
1
$\begingroup$

Training neural networks is a very broad topic. Backpropagation is just the chain rule applied in a clever way to neural networks. So yes, it deals with arbitrary networks as long as they do not have cicles (directed acyclic graphs). It is furthermore assumed that connections go from one layer to the immediately next one. It is still possible to deal with more general architectures.

A nice introductory paper is Neural networks and their applications. The book by the author is, IMHO, the best introductory text to the topic.

Particularly, if you are willing to implement it yourself, there are a number of issues you have to be aware of, which are explained in great detail. Another great resource where you will find lots of advice is the UFLDL Tutorial.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Looks like I have my reading cut out for me for the next month or two. I've since abandoned the project I was working on at the time of posting this question. Having considerably more exposure to a more abstract languages like Lua, I tried constructing C data-structures in a very convoluted and ultimately erroneous manner. Either way, I expect to give ANN's another go in the near future. Thank you for the response! $\endgroup$
    – Jaclyn Brockschmidt
    Dec 22, 2014 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.