11
$\begingroup$

When updating the weights of a neural network using the backpropagation algorithm with a momentum term, should the learning rate be applied to the momentum term as well?

Most of the information I could find about using momentum have the equations looking something like this:

$W_{i}' = W_{i} - \alpha \Delta W_i + \mu \Delta W_{i-1}$

where $\alpha$ is the learning rate, and $\mu$ is the momentum term.

if the $\mu$ term is larger than the $\alpha$ term then in the next iteration the $\Delta W$ from the previous iteration will have a greater influence on the weight than the current one.

Is this the purpose of the momentum term? or should the equation look more like this?

$W_{i}' = W_{i} - \alpha( \Delta W_i + \mu \Delta W_{i-1})$

ie. scaling everything by the learning rate?

$\endgroup$

1 Answer 1

10
$\begingroup$

Using Backpropagation with momentum in a network with $n$ different weights $W_k$ the $i$-th correction for weight $W_k$ is given by

$\Delta W_k(i) = -\alpha \frac{\partial E}{\partial W_k} + \mu \Delta W_k(i-1)$ where $\frac{\partial E}{\partial W_k} $ is the variation of the loss w.r.t. $W_k$.

Introduction of the momentum rate allows the attenuation of oscillations in the gradient descent. The geometric idea behind this idea can probably best be understood in terms of an eigenspace analysis in the linear case. If the ratio between lowest and largest eigenvalue is large then performing a gradient descent is slow even if the learning rate large due to the conditioning of the matrix. The momentum introduces some balancing in the update between the eigenvectors associated to lower and larger eigenvalues.

For more detail I refer to

http://page.mi.fu-berlin.de/rojas/neural/chapter/K8.pdf

$\endgroup$
5
  • $\begingroup$ What does the underbrace signify? $\endgroup$ Oct 13, 2014 at 16:27
  • $\begingroup$ okay, so the momentum term is incorporated when calculating the $\Delta W_k$ term, rather than added on when calculating the "new" weight value? Just to clarify, should your term $\mu W_k(i-1)$ be $\mu \Delta W_k(i-1)$? or is it a proportion of the actual weight rather than the change in rate? thanks for your response, and for the link to the paper. $\endgroup$
    – guskenny83
    Oct 14, 2014 at 3:03
  • $\begingroup$ thank you for pointing out the mistake. It is of course $\Delta W_k(i-1)$ $\endgroup$
    – nico
    Oct 14, 2014 at 6:35
  • $\begingroup$ What do you mean by "variation of the loss"? Is that something like "variation in the error"? $\endgroup$ Apr 10, 2018 at 15:06
  • $\begingroup$ It means nothing but the derivative of the error with respect to weights. $\endgroup$
    – nico
    May 8, 2018 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.