# Is Newton's algorithm really this much better than conjugate gradient descent?

I have a function I'm minimizing. I'm using conjugate gradient descent and the Newton algorithm.

I am experiencing that the Newton algorithm is absurdly faster. Like, it finishes it 5-6 iterations, while the conjugate gradient takes 2000 iterations (and regular gradient descent takes 5000 iterations).

I know what's causing the problem too: the learning rate. In Newton's method, a learning rate of $$\alpha = 1$$ works. But I can't use this in the gradient descent methods, where such a choice of $$\alpha$$ diverges. In these methods, I am forced to use $$\alpha = 0.01$$.

But anyways, 5 iterations vs thousands of iterations is still an absurd difference. Is Newton's method really this good??!

I would caution against drawing a conclusion about the performance of Newton's method on all functions, based on an empirical observation of its performance on one particular function (a sample size of $$n=1$$).