Please note that I am talking in about theory rather than ''what someone would do in a real, practical situation''.

Given a multi-layer Perceptron with at least 1 hidden layer, and sigmoid (or other non-linearities), and a loss function, let it be a quadratic loss function such as $L= ||\phi(X,W)-Y_{target} ||_{2}^2$, where $\phi(X,W)$ is the output of the net, W are the weights, X the data matrix, $Y_{target}$ the correct labels. I think it does not make sense to solve for $\nabla_W L = 0$ to find a minimum ? I think it does not because, first of all, the function $L$, even though being ''the square of something'', is non-convex (possibly very complex with many ''hills'' and ''valleys'') and there can be many different values of $W$ giving a value of 0 for the gradient. Moreover, these points (W) where the gradient $\nabla_W L = 0$ could also be maxima rather than minima! Therefore, I think solving $\nabla_W L = 0$ would only make sense if $L$ is convex.

Can someone with good theoretical background answer this/confirm/complete this please?


closed as off-topic by Discrete lizard Jun 3 at 5:10

  • This question does not appear to be about computer science within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.