3
$\begingroup$

Suppose we have a function $f$,

$$ C = f(A,B), $$

where $A$, $B$ and $C$ are random variables.

I notice that when the random variables are binary ($\{0, 1\}$) and $f$ is the XOR operation, we have the following identity:

$$ H(C|A) = H(B), $$

where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.

Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above holds.

$\endgroup$
  • 1
    $\begingroup$ The function needs to be injective with respect to its second argument. $\endgroup$ – Yuval Filmus Mar 28 at 8:04
  • $\begingroup$ @YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it? $\endgroup$ – hklel Mar 28 at 8:11
4
$\begingroup$

The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.

Let's write your equation in a slightly different way: $$ H(B) = H(f(A,B)|A) = \operatorname*{\mathbb{E}}_{a \sim A} H(f(a,B)). $$ Clearly $H(f(a,B)) \leq H(B)$, with equality if and only if $f(a,b_1) \neq f(a,b_2)$ whenever $b_1 \neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 \neq b_2$, we have $f(a,b_1) \neq f(a,b_2)$.

$\endgroup$
  • 1
    $\begingroup$ $H(f(A,B)|A)=\mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent. $\endgroup$ – xskxzr Mar 28 at 8:45
  • 3
    $\begingroup$ The conclusion that $f$ is injective in the second argument is only correct if $\Pr(A=a)>0$ and $\Pr(B=b)>0$ for all $(a,b)\in\operatorname{dom}(f)$. $\endgroup$ – Emil Jeřábek Mar 28 at 10:03
8
$\begingroup$

Note

\begin{align} 0&=H(C|A,B)\\ &=H(A,B,C)-H(A,B)\\ &=H(B|A,C)+H(C|A)+H(A)-H(A,B)\quad\text{(chain rule)}\\ &=H(B|A,C)+H(C|A)-H(B|A), \end{align}

so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)\ge 0$ and $H(B)\ge H(B|A)$, your condition is equivalently $H(B|A,C)=0\wedge H(B)=H(B|A)$.

For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain $\{b\mid \mathrm{Pr}\{A=a, B=b\}>0\}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.

$\endgroup$
  • 2
    $\begingroup$ The conclusion that $f$ is injective in the second argument is only correct if $\Pr(A=a)>0$ and $\Pr(B=b)>0$ for all $(a,b)\in\operatorname{dom}(f)$. $\endgroup$ – Emil Jeřábek Mar 28 at 10:04
  • $\begingroup$ @EmilJeřábek Thanks, fixed. $\endgroup$ – xskxzr Mar 28 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.