The definition of the entropy is

$$H(Y) = -\sum p(y_j)\log_2 p(y_j)\,.$$

Now my text book says to compute the entropy for each attribute we consider the grouping of the data by that attribute now in each group we calculate the entropy (with respect to classes in each subgroup) and do a weighted sum. In this way the attribute we choose has the most purity of data in each of their subgroups.

Then I thought why cannot we use the entropy in this way:

We consider each class and calculate the entropy of each class (with respect to the groups they belong to) and do a weighted average.

For example, let's say by putting attribute $t_1$ in the root, the data are split in to sets and our class labels are $+$ and $-$.

  group 1         group2 

{30 +   1 -}       {30 +   1 -}

(the numbers show the frequency of each class)

Method 1:

\begin{align*} H(\text{group 1}) &= -(\tfrac1{31} \log\tfrac1{31}+\tfrac{30}{31}\log\tfrac{30}{31})\\ H(\text{group 2}) &= -(\tfrac{1}{31} \log\tfrac1{31}+\tfrac{30}{31}\log\tfrac{30}{31}\qquad \text{(just the same)}\\ H(f1) &= \tfrac{31}{62} H(\text{group 1}) + \tfrac{31}{62} H(\text{group 2})\,. \end{align*}

Method 2:

\begin{align*} H(\text{label }+) &= -(\tfrac{30}{60}\log\frac{30}{60} + \tfrac{30}{60}\log\tfrac{30}{60})\\ H(\text{label }-) &= -(\tfrac12\log\frac12 + \tfrac12\log\tfrac12)\\ H(f1) &= \tfrac{60}{62} H(\text{label }+) + \frac{2}{62} H(\text{label }-)\,. \end{align*}

Now I know that the second method does not measure the purity of each subgroup but it can help us. For example,

f1                               f2

group 1        group 2           group 1      gropu2

{30+  1-}    {30+  1-}           {30+  1-}    {30-  1+}

In this case the method one thinks both of the attributes do a good job of making pure subgroups but we also need the ability to make good decisions meaning differentiating between different class labels with the help of our attribute. f2 does this much better and I think method2 can help us with that.

Here method 1 gives equal entropy while method 2 gives much lower entropy for f2.

But again method2 is not enough by itself since it cannot measure purity.

My question: am I right? If yes then how can we combine the use of both these measures to choose the best attribute for each node?

  • $\begingroup$ I'm confused. If two methods of calculating the entropy of a distribution give different answers, one of them is not calculating the entropy! $\endgroup$ Commented May 12, 2016 at 21:17
  • 1
    $\begingroup$ There are two different measures, but it is unclear what is your end goal, and why you believe one method is better than the other (towards that implied end goal) $\endgroup$
    – Ran G.
    Commented May 12, 2016 at 22:05
  • $\begingroup$ I'am not saying one method is better than the oder . I think we should combine the use of both measures . $\endgroup$
    – KFkf
    Commented May 13, 2016 at 6:34
  • $\begingroup$ And Iam looking for a proper way to do that . $\endgroup$
    – KFkf
    Commented May 13, 2016 at 6:34
  • $\begingroup$ The first method is according to my text book . The second is what i propose we should use because of casese like example 2 . $\endgroup$
    – KFkf
    Commented May 13, 2016 at 6:37


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.