I've seen two versions of the cross entropy cost function, and conflicting information about it. \begin{equation}J(\theta) = -\frac{1}{N} \sum_{n=1}^N\sum_{i=1}^C y_{ni}\log \hat{y}_{n_i} (\theta)\end{equation} $$ C(\theta) = - \frac{1}{N}\sum_{n=1}^N \sum_{i=1}^{C}[ y_{ni}\log (\hat{y}_{ni})+ (1-y_{ni}) \log(1-\hat{y}_{ni})] $$

Some are saying that the second one is equivalent to the first for the case where there are only two classes, which makes sense. But couldn't the second one also be used for more than two classes? For example, say we have three class classification, with the ground truth $\vec{y} = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}$ and $\vec{\hat{y}} =\begin{bmatrix} 0.7 \\ 0.1 \\ 0.2 \end{bmatrix} $. Then with the first equation, cross entropy would simply be $-\log(0.7)$. But couldn't we also use the second equation and calculate cross entropy as $-\log(0.7) - \log(0.9)-\log(0.8)?$ When would we use the first equation and when would we use the second one, and why?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.