# Gradient descent overshoot - why does it diverge?

I understand that it can overshoot the minimum when the learning rate is too large. But I can't understand why it would diverge.
Let's say we have

$$J(\theta_0, \theta_1) = \frac{1}{2m}\sum_{i=1}^m (h_\theta(x^i)-y^i)^2$$

$$\theta_1 := \theta_1-\alpha\frac{\partial}{\partial\theta_1}J(\theta_1)$$

When the slope is negative the cost will converge to the minimum from the left of the graph, as $\theta_1$ will increase.
When the slope is positive it will converge from the right of the graph, as $\theta_1$ will decrease.
Now it might overshoot when the learning rate $\alpha$ is too large.
In that case it should overshoot again.
But not by much, should it not circle around the minimum? Why would it diverge?

• If $\alpha$ is too large it could continually overshoot and not converge. You can dynamically adjust the learning rate by using a simple line search algorithm that satisfies the Wolfe condition for each step. – Nicholas Mancuso Mar 16 '16 at 17:46
• I hope that this might be of some help: medium.com/@prash24goel/… – Prash Goel Jul 6 '19 at 17:44
• Take a look at this. – Rodrigo de Azevedo Aug 8 '19 at 8:52