9
$\begingroup$

When updating the weights of a neural network using the backpropagation algorithm with a momentum term, should the learning rate be applied to the momentum term as well?

Most of the information I could find about using momentum have the equations looking something like this:

$W_{i}' = W_{i} - \alpha \Delta W_i + \mu \Delta W_{i-1}$

where $\alpha$ is the learning rate, and $\mu$ is the momentum term.

if the $\mu$ term is larger than the $\alpha$ term then in the next iteration the $\Delta W$ from the previous iteration will have a greater influence on the weight than the current one.

Is this the purpose of the momentum term? or should the equation look more like this?

$W_{i}' = W_{i} - \alpha( \Delta W_i + \mu \Delta W_{i-1})$

ie. scaling everything by the learning rate?

$\endgroup$
10
$\begingroup$

Using Backpropagation with momentum in a network with $n$ different weights $W_k$ the $i$-th correction for weight $W_k$ is given by

$\Delta W_k(i) = -\alpha \frac{\partial E}{\partial W_k} + \mu \Delta W_k(i-1)$ where $\frac{\partial E}{\partial W_k} $ is the variation of the loss w.r.t. $W_k$.

Introduction of the momentum rate allows the attenuation of oscillations in the gradient descent. The geometric idea behind this idea can probably best be understood in terms of an eigenspace analysis in the linear case. If the ratio between lowest and largest eigenvalue is large then performing a gradient descent is slow even if the learning rate large due to the conditioning of the matrix. The momentum introduces some balancing in the update between the eigenvectors associated to lower and larger eigenvalues.

For more detail I refer to

http://page.mi.fu-berlin.de/rojas/neural/chapter/K8.pdf

$\endgroup$
  • $\begingroup$ What does the underbrace signify? $\endgroup$ – David Richerby Oct 13 '14 at 16:27
  • $\begingroup$ okay, so the momentum term is incorporated when calculating the $\Delta W_k$ term, rather than added on when calculating the "new" weight value? Just to clarify, should your term $\mu W_k(i-1)$ be $\mu \Delta W_k(i-1)$? or is it a proportion of the actual weight rather than the change in rate? thanks for your response, and for the link to the paper. $\endgroup$ – guskenny83 Oct 14 '14 at 3:03
  • $\begingroup$ thank you for pointing out the mistake. It is of course $\Delta W_k(i-1)$ $\endgroup$ – nico Oct 14 '14 at 6:35
  • $\begingroup$ What do you mean by "variation of the loss"? Is that something like "variation in the error"? $\endgroup$ – starbeamrainbowlabs Apr 10 '18 at 15:06
  • $\begingroup$ It means nothing but the derivative of the error with respect to weights. $\endgroup$ – nico May 8 '18 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.