0
$\begingroup$

I am learning quantum computing and as a background study, I am currently learning fundamentals of classical information theory. I thought it best to ask my doubts here. In Nielsen and Chuang, it is stated that mutual information I(X:Y) of two random variables X and Y is the information they have in common while in some books, it is written that mutual information I(X:Y) is the information one variable(say X) has about the other(say Y). I can't understand it intuitively how the two definitions are equivalent. Also, the symmetric property of mutual information,i.e., I(X:Y)=I(Y:X) is obvious to me from the first definition but not from the second one.

$\endgroup$
  • 1
    $\begingroup$ I don't see any definitions in your post. The mutual information between two random variables is given by a formula. Your "definitions" are just interpretations of this formula. $\endgroup$ – Yuval Filmus Apr 1 at 21:17
  • $\begingroup$ What is the motivation behind the formula? Also, how are these two "interpretations" equivalent? $\endgroup$ – Rohan Joshi Apr 2 at 6:24
  • $\begingroup$ You can read Shannon’s 1948 paper, where he introduces mutual information. It is the right quantity to consider in some contexts. $\endgroup$ – Yuval Filmus Apr 2 at 6:26
  • $\begingroup$ How do I prove that S(X)-S(X|Y) = S(Y)-S(Y|X)? $\endgroup$ – Rohan Joshi Apr 2 at 12:16
0
$\begingroup$

You can define the mutual information as $I(X;Y) = H(X) - H(X|Y)$. This definition is symmetric in $X,Y$ since $$ H(X)-H(X|Y) = H(X) - \sum_y \Pr[Y=y] H(X|Y=y) = \\ \sum_x \Pr[X=x] \log \frac{1}{\Pr[X=x]} - \sum_y \Pr[Y=y] \sum_x \Pr[X=x|Y=y] \log \frac{1}{\Pr[X=x|Y=y]} = \\ \sum_{x,y} \Pr[X=x,Y=y] \log \frac{1}{\Pr[X=x]} - \sum_{x,y} \Pr[X=x,Y=y] \log \frac{\Pr[Y=y]}{\Pr[X=x,Y=y]} = \\ \sum_{x,y} \Pr[X=x,Y=y] \log \frac{\Pr[X=x,Y=y]}{\Pr[X=x]\Pr[Y=y]}, $$ and this expression is symmetric in $X,Y$.

This calculation also shows that if $X,Y$ are independent then $I(X;Y) = 0$, since then $\Pr[X=x,Y=y] = \Pr[X=x] \Pr[Y=y]$.

Moreover, if we write $$ I(X;Y) = \sum_{x,y} \Pr[X=x] \Pr[Y=y] \frac{\Pr[X=x,Y=y]}{\Pr[X=x] \Pr[Y=y]} \log \frac{\Pr[X=x,Y=y]}{\Pr[X=x] \Pr[Y=y]}, $$ then the convexity of $z\log z$ implies that $I(X;Y) \geq z\log z$, where $$ z = \sum_{x,y} \Pr[X=x] \Pr[Y=y] \frac{\Pr[X=x,Y=y]}{\Pr[X=x] \Pr[Y=y]} = \sum_{x,y} \Pr[X=x,Y=y] = 1. $$ Thus $I(X;Y) \geq 0$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.