1
$\begingroup$

In the answer to "what is the Hessian" at this site:

https://stackoverflow.com/questions/23297090/how-calculating-hessian-works-for-neural-network-learning

the person answering the question concludes by saying

"Fun fact, adding the momentum term to your gradient based optimization is (under sufficient conditions) approximating the hessian based optimization (and is far less computationally expensive)."

I have tried to understand this point and look online but am not sure what is meant by the momentum approximating hessian based optimization. How is momentum like hessian based optimization?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.