The definition of the entropy is

$$H(Y) = -\sum p(y_j)\log_2 p(y_j)\,.$$

Now my text book says to compute the entropy for each attribute we consider the grouping of the data by that attribute now in each group we calculate the entropy (with respect to classes in each subgroup) and do a weighted sum. In this way the attribute we choose has the most purity of data in each of their subgroups.

Then I thought why cannot we use the entropy in this way:

We consider each class and calculate the entropy of each class (with respect to the groups they belong to) and do a weighted average.

For example, let's say by putting attribute $t_1$ in the root, the data are split in to sets and our class labels are $+$ and $-$.

  group 1         group2 

{30 +   1 -}       {30 +   1 -}

(the numbers show the frequency of each class)

Method 1:

\begin{align*} H(\text{group 1}) &= -(\tfrac1{31} \log\tfrac1{31}+\tfrac{30}{31}\log\tfrac{30}{31})\\ H(\text{group 2}) &= -(\tfrac{1}{31} \log\tfrac1{31}+\tfrac{30}{31}\log\tfrac{30}{31}\qquad \text{(just the same)}\\ H(f1) &= \tfrac{31}{62} H(\text{group 1}) + \tfrac{31}{62} H(\text{group 2})\,. \end{align*}

Method 2:

\begin{align*} H(\text{label }+) &= -(\tfrac{30}{60}\log\frac{30}{60} + \tfrac{30}{60}\log\tfrac{30}{60})\\ H(\text{label }-) &= -(\tfrac12\log\frac12 + \tfrac12\log\tfrac12)\\ H(f1) &= \tfrac{60}{62} H(\text{label }+) + \frac{2}{62} H(\text{label }-)\,. \end{align*}

Now I know that the second method does not measure the purity of each subgroup but it can help us. For example,

f1                               f2

group 1        group 2           group 1      gropu2

{30+  1-}    {30+  1-}           {30+  1-}    {30-  1+}

In this case the method one thinks both of the attributes do a good job of making pure subgroups but we also need the ability to make good decisions meaning differentiating between different class labels with the help of our attribute. f2 does this much better and I think method2 can help us with that.

Here method 1 gives equal entropy while method 2 gives much lower entropy for f2.

But again method2 is not enough by itself since it cannot measure purity.

My question: am I right? If yes then how can we combine the use of both these measures to choose the best attribute for each node?

  • $\begingroup$ I'm confused. If two methods of calculating the entropy of a distribution give different answers, one of them is not calculating the entropy! $\endgroup$ – David Richerby May 12 '16 at 21:17
  • 1
    $\begingroup$ There are two different measures, but it is unclear what is your end goal, and why you believe one method is better than the other (towards that implied end goal) $\endgroup$ – Ran G. May 12 '16 at 22:05
  • $\begingroup$ I'am not saying one method is better than the oder . I think we should combine the use of both measures . $\endgroup$ – kiyarash May 13 '16 at 6:34
  • $\begingroup$ And Iam looking for a proper way to do that . $\endgroup$ – kiyarash May 13 '16 at 6:34
  • $\begingroup$ The first method is according to my text book . The second is what i propose we should use because of casese like example 2 . $\endgroup$ – kiyarash May 13 '16 at 6:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.