How to interpret plot of cost function in gradient descend algorithm?

Question

I have a data set to which I want to apply logistic regression. I set my cost function to be

$J(\theta) = -\frac{1}{m} [\sum\limits_{i=1}^{m} y^{(i)} log{(h_\theta \cdot x^{(i)})} + (1-y^{(i)}) log(1-h_\theta(x^{(i)}))$

and I am running gradient descend algorithm.

When I plot the cost function as a function of the number of iterations, I am getting a plot like this

Which I think is a good sign.

However, when I zoom in on the initial portion of the curve, I get this plot

I don't know what to make out of the rising and falling in the cost in the beginning. Is that something that we can expect? If so, why? Shouldn't the gradient descend algorithm decrease the cost after every step? How is it that after some iterations there is an increase in the cost? I know that it is possible for gradient decent not to converge if the learning rate is big enough but the algorithm does seem to converge eventually. So why is it that there are some instances where gradient descend tends to increase the cost?

Answer 1

I don't know what to make out of the rising and falling in the cost in the beginning. Is that something that we can expect? If so, why? Shouldn't the gradient descend algorithm decrease the cost after every step?

Gradient descent only minimizes the cost if the learning rate is low enough. Also, in practice most of the time you don't use gradient descent on the whole dataset, but on a mini-batch. This means you don't get an guarantee that the global cost is falling for a single step.

An analogy which is often used is that of a landscape with hills. The landscape is your cost. Each point in the landscape is a specific configuration of weights. The gradient shows in the direction of steepest descent. That might be a rabbit hole on a hill, but it might also be downwards to the valley. Now, imagine it shows to the rabbit hole. Instead of going down the rabbit hole, you make a bigger step and so you go up the hill. But then you are on a different position. For your next step, you come away from the rabbit hole and get towards a better (local) minimum.

As always when it comes to training, I would like to point out the awesome gifs by Alec Redford: Gifs 1, and Gifs 2