Entropy and Information Gain for multi attributes

Question

Is there any way that for specific dataset I can measure entropy and information gain for two or more attributes together? Let's say we have the following dataset:

$\begin{array}{cccccc|c} x1 & x2 & x3 & x4 & x5 & x6 & y \\ \hline 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 1 & 0 & 1 & 1 & 1\\ 0 & 1 & 0 & 1 & 1 & 1 & 1\\ 0 & 1 & 1 & 0 & 0 & 1 & 0\\ 0 & 1 & 1 & 1 & 0 & 0 & 1\\ 1 & 0 & 0 & 1 & 1 & 0 & 0\\ 1 & 0 & 1 & 1 & 0 & 1 & 1\\ 1 & 1 & 1 & 0 & 0 & 0 & 1\\ 1 & 1 & 1 & 1 & 1 & 1 & 1\\ \end{array}$

How can I calculate the information gain for $x1,x2$ or $x1,x2,x3$ or $x1,x2,x3,x4$

Answer 1

You can group attributes together, but (assuming you're building some kind of decision tree) your decision tree will no longer be a binary tree, and you probably won't be any better off than just splitting by one attribute. Say for example you group $x_1$ and $x_2$ together, then there are 4 possible values for this new meta-attribute. You can use the same entropy calculation for this meta-attribute, and split on these four values.

But ask yourself this: why would you want to split on multiple attributes at once? Sure, it's true that $y=1$ whenever $x_1$, $x_2$, and $x_3$ are 1, but once you split on this you can't split on any subset of these three (for example, observe that $y=0$ whenever $x_2$ and $x_3$ are zero. If we split on all three variables we'd need two identical subtrees, one for $x_1=1, x_2=0, x_3=0$ and one for when they're all zero.)