I came across a book where the author uses the following property of mutual information:

Let $X$,$Y$,$Z$ be arbitrary discrete random variables and let $W$ be an indicator random variable.

$$ (1)\ \ I[ X : Y \mid Z ] = Pr(W=0) I[ X : Y \mid Z, W=0 ] + Pr(W=1) I[ X : Y \mid Z, W=1 ]\ $$

I don't understand why this property holds in general. To show this I was thinking to proceed as follows: \begin{align} I[ X : Y \mid Z ] &= E_z[ I[ X : Y \mid Z = z ]] \\ &= E_w[ E_z[ I[ X: Y \mid Z = z]\ |\ W=w ] ] \\ &= Pr(W=0)E_z[ I[ X: Y \mid Z = z]\ |\ W=0 ] \\ &+ Pr(W=1)E_z[ I[ X: Y \mid Z = z]\ |\ W=1 ]. \end{align} where the second line follows by the law of total expectation. However, this does not seem to be the right approach since it's not clear to me that $$ E_z[ I[ X: Y \mid Z = z]\ |\ W=w ] = I[ X : Y \mid Z, W=w ]$$ holds.

What is the right way to show (1)?

  • $\begingroup$ Are you sure you wrote the exact equation with no further assumptions? (1) reads as I[X:Y|Z] = I[X:Y|Z,W] which might not hold in some cases. $\endgroup$ – Ran G. Feb 17 at 7:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.