I'm thinking about gradient descent, but I don't get it.
I understand that it can overshoot the minimum when the learning rate is too large. But I can't understand why it would diverge.
Let's say we have

$$J(\theta_0, \theta_1) = \frac{1}{2m}\sum_{i=1}^m (h_\theta(x^i)-y^i)^2$$

$$\theta_1 := \theta_1-\alpha\frac{\partial}{\partial\theta_1}J(\theta_1)$$

When the slope is negative the cost will converge to the minimum from the left of the graph, as $\theta_1$ will increase.
When the slope is positive it will converge from the right of the graph, as $\theta_1$ will decrease.
Now it might overshoot when the learning rate $\alpha$ is too large.
In that case it should overshoot again.
But not by much, should it not circle around the minimum? Why would it diverge?

  • 2
    $\begingroup$ If $\alpha$ is too large it could continually overshoot and not converge. You can dynamically adjust the learning rate by using a simple line search algorithm that satisfies the Wolfe condition for each step. $\endgroup$ – Nicholas Mancuso Mar 16 '16 at 17:46
  • $\begingroup$ I hope that this might be of some help: medium.com/@prash24goel/… $\endgroup$ – Prash Goel Jul 6 at 17:44
  • $\begingroup$ Take a look at this. $\endgroup$ – Rodrigo de Azevedo Aug 8 at 8:52

If you overshoot by 10% you are fine, for example if the solution is 0 and the starting value 1000, you go to -100, +10, -1, +0.1 etc. If you overshoot by 80%, you converge but much smaller. If you overshoot by 110% you go from 1,000 to -1,100, +1,210, -1,331 etc. - divergence.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.