Suppose we have a 2-layer neural network completely connected with $d^{(0)}$ input units, $d^{(1)}$ hidden units and $d^{(2)}$ output units. We consider the error function given by $J(w) = \frac{1}{2}\sum_{k=1}^{d^{(2)}}(t_k-c_k)^2$, where the vector $t = (t_1,\dots,t_{d^{(2)}})$ represents the labels and $z = (z_1,\dots,z_{d^{(2)}})$ represents the values obtained by the network. Suppose that the inputs to the second layer are calculated as $z_k = \sum_{j=0}^{d^{(2)}}y_j w_{kj}$, where $y$ represents the hidden layer output.
I want to know which are the learning rules between hidden and output layer, and between input and hidden layer, for a given activation function $\theta$.
Based on SGD, the learning rule consists in adding to the weights the gradient ot the error function.
For the first one I have tried this. I need to compute the gradient, so I derive $J(w)$ respect to the weight $w_{kj}$ and apply the chain rule:
$$\frac{\partial}{\partial w_{kj}}J(w) = \frac{\partial(\frac{1}{2}(t_k-z_k)^2)}{\partial w_{kj}}=\frac{\partial(\frac{1}{2}(t_k-z_k)^2)}{\partial z_k}\frac{\partial z_k}{\partial w_{kj}} = -(t_k-z_k) \frac{\partial (y_jw_{kj})}{\partial w_{kj}} $$
But I'm not sure about how to continue. If $y_j$ is not dependent of the weight $w_{kj}$, then the derivative turns into $-(t_k-z_k)y_j$, but is this right? Would this be the learning rule for the weights between hidden and output layer?
And for the learning rule between input and hidden layer I don't know how should I start. Any idea would be appreciated. Thanks.
