Correct cost function of multi classification problem using neural network?

Question

I am going through machine learning course on coursera. While going through the section on neural networks I came across the cost function for multi - classification problem using neural networks ( ignoring the regularization term ): $$\begin{gather*} J(\Theta) = - \frac{1}{m} \sum_{i=1}^m \sum_{k=1}^K \left[y^{(i)}_k \log ((h_\Theta (x^{(i)}))_k) + (1 - y^{(i)}_k)\log (1 - (h_\Theta(x^{(i)}))_k)\right] \end{gather*}$$

where $m$ is the number of training data points $\{(x_1,y_1),...(x_m,y_m)\}$. Each data point is classified into one of the $K$ categories ie. $1\;\le{\;y_i}\;\le{K}$ ( so $y_k^{(i)}$ means if $\;y_k^{(i)} == k\;$ ). And $h_\Theta$ is the hypothesis function.

My doubt is why does the cost function has a summation over $K$. Why aren't there $K$ separate cost functions ( $ 1 \; \le{k} \le K$ ), and we have different $\Theta$ when we minimize each of the $K$ cost functions separately:

$$\begin{gather*} J(\Theta)_k = - \frac{1}{m} \sum_{i=1}^m \left[y^{(i)}_k \log ((h_\Theta (x^{(i)}))_k) + (1 - y^{(i)}_k)\log (1 - (h_\Theta(x^{(i)}))_k)\right] \end{gather*}$$

In the second approach we are trying one-vs-all classification. How is it different from first approach. First approach is taking sum over all cost functions of second approach. So minimization problem being solved in both cases are different. But I am unable to pin point what the difference between the two approaches is ? Both seem intuitively correct to me. Am I doing some stupid mistake ?

Answer 1

The loss function you have shown is not correct. The correct one is

$$\begin{gather*} J(\Theta) = - \frac{1}{m} \sum_{i=1}^m \sum_{k=1}^K y^{(i)}_k \log ((h_\Theta (x^{(i)}))_k) \end{gather*}$$

The loss function has a summation over $K$ because we must take into account all possible classes. The neural network outputs $K$ values, $h_{\Theta}(x^{(i)})$, and the loss function should depend on all $K$ of those. There is a single loss function because we are training a single neural network, and we need exactly one loss function per neural network; if you wanted to train $K$ neural networks, then you would use $K$ loss functions.

Define $z^{(i)}$ to be the class of $x^{(i)}$ according to the training set, i.e., we take $z^{(i)}$ to be the value of $k$ such that $y^{(i)}_k=1$. Then the loss function above is equivalent to

$$\begin{gather*} J(\Theta) = - \frac{1}{m} \sum_{i=1}^m \log ((h_\Theta (x^{(i)}))_{z^{(i)}}) \end{gather*}$$

which does not have a summation over $K$.

I suggest reading more about the cross-entropy loss. See, e.g., https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/

There are some errors in your notation. The training set has points $(x^{(i)},y^{(i)})$, not $x_i,y_i$. Here $y^{(i)}_k$ means that $y^{(i)}=k$, not that $y^{(i)}_k=k$ (the latter would be circular, defining $y^{(i)}_k$ in terms of itself).