# Formal definition of loss surface of multi-layered networks

Let $$\mathcal{L}$$ be a loss function associated with a multi-layered neural network. So it seems almost everyone in AI/ML community is interested in the Hessian $$H=\partial^2 \mathcal{L}$$ of $$\mathcal{L}$$. So if you read [1] on page 193 the left colum you find the following text

"An important question concerns the distribution of critical points (maxima, minima, and saddle points) of such functions. Results from random matrix the- ory applied to spherical spin glasses have shown that these functions have a combinatorially large number of saddle points. Loss surfaces for large neural nets have many local minima that are essentially equivalent from the point of view of the test error, and these minima tend to be highly degenerate, with many eigenvalues of the Hessian near zero. " Could anyone in the community shed some light on how formally the authors define (perhaps assuming some accepted definition in the community) the loss surface associated with either neural network or the loss function of the neural network?

My guess is that it is the level set of the loss function attaining one of its critical values (of the loss function). I would appreciate if someone would share any knowledge on the nature of the formal definition of the loss surface or any accepted common knowledge definition. Many thanks.

[1] A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, pages 192–204, 2015.

In the empirical risk minimization framework, we let $$w$$ denote the weights of the neural network (which we are trying to learn) and define $$L(w)$$ to be the total loss (across the entire training set) if you use weights $$w$$. Then, the optimization problem is to find $$w$$ that minimizes $$L(w)$$.
"Loss surface" refers to the graph of this function. $$L$$ is multidimensional so we obtain a graph in some high-dimensional space.
• @user88903, by "surface", they are just referring to the graph of this function ($L$ is multidimensional so it yields a surface). See, e.g., en.wikipedia.org/wiki/…. – D.W. Jul 5 '18 at 15:38