6
$\begingroup$

I am reading a book about Neural Networks, that explains the backpropagation algorithm. It explains that the error function is the sum of error of all inputs, and that the algorithm minimizes this sum, which makes perfect sense to me, but than in goes on stating that to achieve that the gradient is calculated for each input, and then summed up. This last step is not understable to me.

Is it implying that the gradient of the sum of all errors (from all inputs) is equal to sum of gradients (from all inputs), that is - that the sum of gradients equals the gradient of sum?

This seems to me like an non trivial step, but it is not explained in the book.

$\endgroup$
  • $\begingroup$ This is a basic property of the gradient, namely linearity: $\nabla (\alpha f + \beta g) = \alpha \nabla f + \beta \nabla g$. $\endgroup$ – Yuval Filmus Sep 18 '15 at 15:08
  • $\begingroup$ right, thanks. somehow this linearity was counterintuitive to me. $\endgroup$ – oded wolff Sep 18 '15 at 16:54
5
$\begingroup$

The gradient, just like the derivative, is linear: for functions $f,g$ on the same variables and scalars $\alpha,\beta$: $$ \nabla(\alpha f + \beta g) = \alpha \nabla f + \beta \nabla g. $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.