1
$\begingroup$

Please note that I am talking in about theory rather than ''what someone would do in a real, practical situation''.

Given a multi-layer Perceptron with at least 1 hidden layer, and sigmoid (or other non-linearities), and a loss function, let it be a quadratic loss function such as $L= ||\phi(X,W)-Y_{target} ||_{2}^2$, where $\phi(X,W)$ is the output of the net, W are the weights, X the data matrix, $Y_{target}$ the correct labels. I think it does not make sense to solve for $\nabla_W L = 0$ to find a minimum ? I think it does not because, first of all, the function $L$, even though being ''the square of something'', is non-convex (possibly very complex with many ''hills'' and ''valleys'') and there can be many different values of $W$ giving a value of 0 for the gradient. Moreover, these points (W) where the gradient $\nabla_W L = 0$ could also be maxima rather than minima! Therefore, I think solving $\nabla_W L = 0$ would only make sense if $L$ is convex.

Can someone with good theoretical background answer this/confirm/complete this please?

$\endgroup$