Given a simple artificial neural network with 1 hidden layer, I want to compute the analytic gradient, to gain a better understanding.

Using a simple loss function L such as:

$L=(1/N)\sum_{k=1}^N|| \textbf{W}_2*g(\textbf{W}_1*\vec{x}_k) - \vec{y}_{target} ||^2_2 $

where W1 is the weight matrix of the hidden layer $h$, W2 is the weight matrix of $h$ and the output layer. The vector $\vec{x}_k$ is the $k_{th}$ training example. The symbol '*' means ordinary matrix multiply here. $g$ is the sigmoid function : $1/(1+exp(-x))$. (The loss function averages over all training examples).

The problem for me is that is seems difficult to derive the gradient with respect to '' all the weights i.e. W1 AND W2 ''.

I can imagine to compute (separately!):

$\partial L / \partial \textbf{W}_1$


$\partial L / \partial \textbf{W}_2$

but not both at the same time...

What am I missing? Can someone clarify?

P.S. I am aware of this beautiful tool: http://www.matrixcalculus.org/

  • $\begingroup$ There was an answer to this question yesterday... did someone remove it? (this would be terrible...) $\endgroup$ – Machupicchu Jun 1 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.