0
$\begingroup$

Is there any way that for specific dataset I can measure entropy and information gain for two or more attributes together? Let's say we have the following dataset:

$\begin{array}{cccccc|c} x1 & x2 & x3 & x4 & x5 & x6 & y \\ \hline 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 1 & 0 & 1 & 1 & 1\\ 0 & 1 & 0 & 1 & 1 & 1 & 1\\ 0 & 1 & 1 & 0 & 0 & 1 & 0\\ 0 & 1 & 1 & 1 & 0 & 0 & 1\\ 1 & 0 & 0 & 1 & 1 & 0 & 0\\ 1 & 0 & 1 & 1 & 0 & 1 & 1\\ 1 & 1 & 1 & 0 & 0 & 0 & 1\\ 1 & 1 & 1 & 1 & 1 & 1 & 1\\ \end{array}$

How can I calculate the information gain for $x1,x2$ or $x1,x2,x3$ or $x1,x2,x3,x4$

$\endgroup$
2
$\begingroup$

You can group attributes together, but (assuming you're building some kind of decision tree) your decision tree will no longer be a binary tree, and you probably won't be any better off than just splitting by one attribute. Say for example you group $x_1$ and $x_2$ together, then there are 4 possible values for this new meta-attribute. You can use the same entropy calculation for this meta-attribute, and split on these four values.

But ask yourself this: why would you want to split on multiple attributes at once? Sure, it's true that $y=1$ whenever $x_1$, $x_2$, and $x_3$ are 1, but once you split on this you can't split on any subset of these three (for example, observe that $y=0$ whenever $x_2$ and $x_3$ are zero. If we split on all three variables we'd need two identical subtrees, one for $x_1=1, x_2=0, x_3=0$ and one for when they're all zero.)

$\endgroup$
  • $\begingroup$ My point is to try and create groups or not binary decision trees. I want to try in these maximums here, for example, $x1,x2,x3$ together. Can you help me to calculate the entropy of three of them together as well information gain? $\endgroup$ – J. Doe Mar 27 '18 at 13:56
  • $\begingroup$ @J.Doe, Use the definition: en.wikipedia.org/wiki/Information_gain_in_decision_trees. Just plug in -- the definition works when there are multiple values. It's not clear why you are having difficulties, so it's not clear how to help you further. If you are still stuck, I suggest editing the question to show your thoughts, what you do understand, and what specifically you are stuck on. $\endgroup$ – D.W. Mar 27 '18 at 15:08
  • $\begingroup$ So, I start with entropy of entire set which is $E(Y) = -\frac{3}{9}*log_{2}(\frac{3}{9})+\frac{6}{9}*log_{2}(\frac{6}{9})=0.91$, then If I want to find the Gain for $X1$ I calculate $E(X1=0)=0.97$ and $E(X1=1)=0.81$ then the Gain is $G(X1) =E(Y)-\frac{5}{9}*E(X1=0)-\frac{4}{9}*E(X1=1)=0.01$ So how should I do the Gain for two or more, example: $G(X1,X2)$ $\endgroup$ – J. Doe Mar 27 '18 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.