Hessian in reinforcement learning

The Hessian of multi-layered network exhibits known behaviour at critical points as shown in [1]. The tools of random matrix theory allow [2] to deduce the asymptotic distribution of the eigenvalues of the Hessian.

There are some numerical experiments with distributed Newton type stochastic methods. The scalability (ability to perform computations in parallel architecture ) is achieved under certain assumptions on the eigenvalues. I am just wondering about two questions:

1) Can one describe the eigenvalues distribution for the Hessian obtained from average reward function in reinforcement learning?

2) If yes, are there any scalable iterative methods (apart from SGD) that can optimise the average reward function and exploit the structure of the eigenvalues of the hessian if possible.

I beg my apology for possible misleading formulations. Please feel free to express your comments/concerns to me and I will edit the post accordingly.

Many thanks.

[1] L. Sagun, L. Bottou, and Y. LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476, 2016.

[2] J. Pennington and Y. Bahri. Geometry of neural network loss surfaces via random matrix theory. In International Conference on Machine Learning, pages 2798–2806, 2017.

• Today, most neural networks are large: e.g., hundreds of thousands or millions of weights. Let $n$ denote the number of such weights. Then the Hessian involves $n^2$ different values. Consequently, when $n$ is in the hundreds of thousands or millions, it is infeasible to compute the Hessian. So, while the Hessian is of conceptual interest in understanding what is going on during the optimization, people don't actually compute the Hessian when they are training neural networks, because that would be infeasibly hard. – D.W. Jul 5 '18 at 15:37
• Dear D.W, I was rather referring to Hessian in reinforcement learning. The average return function is optimised with respect to policy parameters. I was hoping to gain understanding if policy parameters form a surface. How the Hessian of the average return function looks like ? – user88903 Jul 5 '18 at 15:53
• @ D.W., You are right about the infeasibility, but I am not referring to any actual computation. It might be possible to avoid unnecessary training by using theoretical understanding. – user88903 Jan 22 at 21:32