1
$\begingroup$

I have a data set to which I want to apply logistic regression. I set my cost function to be

$J(\theta) = -\frac{1}{m} [\sum\limits_{i=1}^{m} y^{(i)} log{(h_\theta \cdot x^{(i)})} + (1-y^{(i)}) log(1-h_\theta(x^{(i)}))$

and I am running gradient descend algorithm.

When I plot the cost function as a function of the number of iterations, I am getting a plot like this

plot 1

Which I think is a good sign.

However, when I zoom in on the initial portion of the curve, I get this plot

plot 2

I don't know what to make out of the rising and falling in the cost in the beginning. Is that something that we can expect? If so, why? Shouldn't the gradient descend algorithm decrease the cost after every step? How is it that after some iterations there is an increase in the cost? I know that it is possible for gradient decent not to converge if the learning rate is big enough but the algorithm does seem to converge eventually. So why is it that there are some instances where gradient descend tends to increase the cost?

$\endgroup$
  • 1
    $\begingroup$ There might be some numerical issues. I wouldn't worry too much. Also, gradient descent is only guaranteed to decrease the function if you take infinitesimal steps (or if the objective function is linear). $\endgroup$ – Yuval Filmus Apr 22 '17 at 15:26
  • $\begingroup$ Please take a look at gradient descent optimization. Yes, the fluctuations are expected, otherwise the step is too small and the number of iterations greatly increases (or the data is somewhat special). $\endgroup$ – Evil Apr 22 '17 at 16:23
3
$\begingroup$

I don't know what to make out of the rising and falling in the cost in the beginning. Is that something that we can expect? If so, why? Shouldn't the gradient descend algorithm decrease the cost after every step?

Gradient descent only minimizes the cost if the learning rate is low enough. Also, in practice most of the time you don't use gradient descent on the whole dataset, but on a mini-batch. This means you don't get an guarantee that the global cost is falling for a single step.

An analogy which is often used is that of a landscape with hills. The landscape is your cost. Each point in the landscape is a specific configuration of weights. The gradient shows in the direction of steepest descent. That might be a rabbit hole on a hill, but it might also be downwards to the valley. Now, imagine it shows to the rabbit hole. Instead of going down the rabbit hole, you make a bigger step and so you go up the hill. But then you are on a different position. For your next step, you come away from the rabbit hole and get towards a better (local) minimum.

As always when it comes to training, I would like to point out the awesome gifs by Alec Redford: Gifs 1, and Gifs 2

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.