Let $\mathcal{L}$ be a loss function associated with a multi-layered neural network. So it seems almost everyone in AI/ML community is interested in the Hessian $H=\partial^2 \mathcal{L}$ of $\mathcal{L}$. So if you read [1] on page 193 the left colum you find the following text

"An important question concerns the distribution of critical points (maxima, minima, and saddle points) of such functions. Results from random matrix the- ory applied to spherical spin glasses have shown that these functions have a combinatorially large number of saddle points. Loss surfaces for large neural nets have many local minima that are essentially equivalent from the point of view of the test error, and these minima tend to be highly degenerate, with many eigenvalues of the Hessian near zero. " Could anyone in the community shed some light on how formally the authors define (perhaps assuming some accepted definition in the community) the loss surface associated with either neural network or the loss function of the neural network?

My guess is that it is the level set of the loss function attaining one of its critical values (of the loss function). I would appreciate if someone would share any knowledge on the nature of the formal definition of the loss surface or any accepted common knowledge definition. Many thanks.

[1] A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, pages 192–204, 2015.


In the empirical risk minimization framework, we let $w$ denote the weights of the neural network (which we are trying to learn) and define $L(w)$ to be the total loss (across the entire training set) if you use weights $w$. Then, the optimization problem is to find $w$ that minimizes $L(w)$.

This is the function that the quote is talking about.

"Loss surface" refers to the graph of this function. $L$ is multidimensional so we obtain a graph in some high-dimensional space.

  • $\begingroup$ Dear D.W. Thanks for your answer. The clarity of the papers allows to deduce that. I was rather concerned about the surface part. Are all this weights form a surface or something else? $\endgroup$ – user88903 Jul 5 '18 at 15:37
  • $\begingroup$ @user88903, by "surface", they are just referring to the graph of this function ($L$ is multidimensional so it yields a surface). See, e.g., en.wikipedia.org/wiki/…. $\endgroup$ – D.W. Jul 5 '18 at 15:38
  • $\begingroup$ Dear D.W. Would it be of any practical utility to use the geometrical information about this surface to escape saddle points while training neural network? $\endgroup$ – user88903 Jul 5 '18 at 15:42
  • $\begingroup$ @user88903, as I explained elsewhere, it is not computationally feasible to compute the Hessian (which seems to be the "geometric information" you are referring to), so it's not feasible to use that to escape saddle points. $\endgroup$ – D.W. Jul 5 '18 at 16:00
  • $\begingroup$ Dear D.W. In that comment, I was asking about the Hessian of the cumulative reward function in reinforcement learning. I would appreciate it if you could give some information on eigenvalue distributions of these Hessians. Many thanks. $\endgroup$ – user88903 Jul 5 '18 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.