# Entropy and Information Gain for multi attributes

Is there any way that for specific dataset I can measure entropy and information gain for two or more attributes together? Let's say we have the following dataset:

$\begin{array}{cccccc|c} x1 & x2 & x3 & x4 & x5 & x6 & y \\ \hline 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 1 & 0 & 1 & 1 & 1\\ 0 & 1 & 0 & 1 & 1 & 1 & 1\\ 0 & 1 & 1 & 0 & 0 & 1 & 0\\ 0 & 1 & 1 & 1 & 0 & 0 & 1\\ 1 & 0 & 0 & 1 & 1 & 0 & 0\\ 1 & 0 & 1 & 1 & 0 & 1 & 1\\ 1 & 1 & 1 & 0 & 0 & 0 & 1\\ 1 & 1 & 1 & 1 & 1 & 1 & 1\\ \end{array}$

How can I calculate the information gain for $x1,x2$ or $x1,x2,x3$ or $x1,x2,x3,x4$

You can group attributes together, but (assuming you're building some kind of decision tree) your decision tree will no longer be a binary tree, and you probably won't be any better off than just splitting by one attribute. Say for example you group $x_1$ and $x_2$ together, then there are 4 possible values for this new meta-attribute. You can use the same entropy calculation for this meta-attribute, and split on these four values.
But ask yourself this: why would you want to split on multiple attributes at once? Sure, it's true that $y=1$ whenever $x_1$, $x_2$, and $x_3$ are 1, but once you split on this you can't split on any subset of these three (for example, observe that $y=0$ whenever $x_2$ and $x_3$ are zero. If we split on all three variables we'd need two identical subtrees, one for $x_1=1, x_2=0, x_3=0$ and one for when they're all zero.)
• My point is to try and create groups or not binary decision trees. I want to try in these maximums here, for example, $x1,x2,x3$ together. Can you help me to calculate the entropy of three of them together as well information gain? – J. Doe Mar 27 '18 at 13:56
• So, I start with entropy of entire set which is $E(Y) = -\frac{3}{9}*log_{2}(\frac{3}{9})+\frac{6}{9}*log_{2}(\frac{6}{9})=0.91$, then If I want to find the Gain for $X1$ I calculate $E(X1=0)=0.97$ and $E(X1=1)=0.81$ then the Gain is $G(X1) =E(Y)-\frac{5}{9}*E(X1=0)-\frac{4}{9}*E(X1=1)=0.01$ So how should I do the Gain for two or more, example: $G(X1,X2)$ – J. Doe Mar 27 '18 at 16:11