Why does the neural network logistic regression cost function sum for all layers only for lambda?

Question

I'm taking Andrew Ng's machine learning course and week 5 covers the training of neural networks. The modified cost function for neural network training is derived from the logistic regression cost function, and is described as follows:

$$\begin{gather*} J(\Theta) = - \frac{1}{m} \sum_{i=1}^m \sum_{k=1}^K \left[y^{(i)}_k \log ((h_\Theta (x^{(i)}))_k) + (1 - y^{(i)}_k)\log (1 - (h_\Theta(x^{(i)}))_k)\right] + \frac{\lambda}{2m}\sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}} ( \Theta_{j,i}^{(l)})^2\end{gather*}$$

Here, K is the number of output units, L is the total number of layers in the network, and ${s_l}$ is the number of units in layer l.

I don't understand why the second half of the cost function, intended to prevent over-fitting by minimizing the values of the theta parameters, is summed across the entire network while the first half of the equation which actually determines theta is only for a specific layer. If each layer's cost function already includes a minimization parameter for that layer's theta values, why is it necessary to perform this minimization globally for every layer in the cost function J?

As I understand it, in training a neural network you mainly treat each layer as completely separate from the next (and in fact, that is one of the major selling points of the NN approach, wherein solving L layers independently of one-another lets you use a simple approach to obtain powerful results) - so why are we summing the second half of the equation over all layers?

Answer 1

The first term depends on the output of the entire network, i.e., on all the parameters for all layers, not just a single layer's parameters.

This is an application of the empirical risk minimization framework for training a classifier, with regularization. The cost function $J(\Theta)$ has the form

$$J(\Theta) = \text{empirical risk} + \text{regularization penalty}.$$

Denote the empirical risk by $R(\Theta)$; it depends on the parameters $\Theta$ (for all layers). Denote the regularization penalty by $P(\Theta)$; it depends on the parameters $\Theta$ (for all layers).

The empirical risk $R(\Theta)$ is the sum of the loss for each instance in the training set, i.e., it has the form

$$R(\Theta) = {1 \over m} \sum_{i=1}^m \ell(h_\Theta(x^{(i)}), y^{(i)}),$$

where $\ell$ is a loss function, $x^{(i)},y^{(i)}$ is the $i$th instance in the training set, and $h_\Theta(x)$ denotes the output of the classifier on input $x$ when the parameters are $\Theta$. Note that the output of the classifier depends on all the parameters for all the layers, so $R(\Theta)$ depends on all the parameters for all layers (not just a single layer).

A comment on notation: We use $\Theta$ to denote all of the parameters, i.e., a single vector that concatenates the parameters for each layer. Here it appears $\Theta^{(l)}$ has been used to denote the parameters for the $l$th layer, so $\Theta = (\Theta^{(1)}, \dots, \Theta^{(L)})$.

In your specific example, we use the cross-entropy loss for $\ell$ (this is also known as the logistic loss function, where there are only two classes). That's how we get the first term.

Also the regularization penalty $P(\Theta)$ depends on all of the parameters, for all layers; it is a sum of penalties for each layer.

If you work through the math, you'll see how we get something of the form you've shown in the question. As this answer hopefully makes clear, each term depends on the parameters for all the layers, not just a single layer.