# Why is the Shannon entropy 0.94 in this example?

Suppose I have a decision tree in which there is a label $L$ under which is the attribute $A$ as shown below. I am given that the Shannon entropy of label $L$ is $H(L) = 0.95$.

I must find the Shannon entropy of $L$ given $A$ ($H(L \mid A)$). Here's what I have tried.

\begin{eqnarray} H(L \mid A) &=& -(\frac{6}{8} \log_2 \frac{4}{6} + \frac{2}{8} \log_2 \frac{1}{2}) \\ &\approx& 0.69 \end{eqnarray}

However, $H(L \mid A) \approx 0.94$. Where did I err? Is my formula for Shannon entropy accurate?

-
The Shannon entropy is always non-negative. You must have got something wrong. – Yuval Filmus Feb 6 '13 at 2:40
I fixed my question. The problem still remains though. Thanks! – David Faux Feb 6 '13 at 2:48
You could also explain what that formula is. That would make the question easier and quicker to understand. – Juho Feb 6 '13 at 2:54
Good point. Sorry about not providing background on the formula. en.wikipedia.org/wiki/Entropy_(information_theory)#Rationale – David Faux Feb 6 '13 at 3:32

Back to the definitions: $$H(L\mid A) = \sum_a p(A=a) H(L \mid A=a).$$

As you compute, $P(A=true)=6/8$ and $P(A=false)=2/8$.

However, you don't compute $H(L\mid A=true)$ but instead compute $P(L=positive\mid A=true)$. [and the same for $A=false$.].

With standard definition of $H()$ we get,

$$H(L\mid A=true) = - 4/6\log_2(4/6) - 2/6\log_2(2/6) = 0.9182958$$ $$H(L\mid A=false) = H(1/2) = 1$$

And thus, $H(L\mid A) = 6/8 \times 0.918 + 2/8\times 1 = 0.938 \approx 0.94$.

-