Suppose we have a 2-layer neural network completely connected with $d^{(0)}$ input units, $d^{(1)}$ hidden units and $d^{(2)}$ output units. We consider the error function given by $J(w) = \frac{1}{2}\sum_{k=1}^{d^{(2)}}(t_k-c_k)^2$, where the vector $t = (t_1,\dots,t_{d^{(2)}})$ represents the labels and $z = (z_1,\dots,z_{d^{(2)}})$ represents the values obtained by the network. Suppose that the inputs to the second layer are calculated as $z_k = \sum_{j=0}^{d^{(2)}}y_j w_{kj}$, where $y$ represents the hidden layer output.

I want to know which are the learning rules between hidden and output layer, and between input and hidden layer, for a given activation function $\theta$.

Based on SGD, the learning rule consists in adding to the weights the gradient ot the error function.

For the first one I have tried this. I need to compute the gradient, so I derive $J(w)$ respect to the weight $w_{kj}$ and apply the chain rule:

$$\frac{\partial}{\partial w_{kj}}J(w) = \frac{\partial(\frac{1}{2}(t_k-z_k)^2)}{\partial w_{kj}}=\frac{\partial(\frac{1}{2}(t_k-z_k)^2)}{\partial z_k}\frac{\partial z_k}{\partial w_{kj}} = -(t_k-z_k) \frac{\partial (y_jw_{kj})}{\partial w_{kj}} $$

But I'm not sure about how to continue. If $y_j$ is not dependent of the weight $w_{kj}$, then the derivative turns into $-(t_k-z_k)y_j$, but is this right? Would this be the learning rule for the weights between hidden and output layer?

And for the learning rule between input and hidden layer I don't know how should I start. Any idea would be appreciated. Thanks.


Have a look at backpropagation, which is essentially what you're trying to do here. Wikipedia has a good derivation that follows exactly your example. If you're more into blogs (and who isn't?) have a look at this one, which goes through the steps of backprop in detail.

Best of luck!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.