5
$\begingroup$

I am learning about information theory and mutual information. However, I am quite confused with MI(Mutual information) vs. PMI(Pointwise mutual information) especially signs of MI and PMI values. Here are my questions.

  • Is MI values a non-negative value or it can be either positive or negative? If it is always a non-negative value, why is it ?

  • As I search online, the PMI can be positive or negative values and the MI is the expected value of all possible PMI. However, expected value can be positive or negative. If MI is really the expected value of PMI, why is it always positive ?

Did I misunderstand anything of MI and PMI here ? Thank you very much,

$\endgroup$
4
$\begingroup$

PMI can clearly be negative, since if $p(x,y)<p(x)p(y)$, then taking $\log \frac{p(x,y)}{p(x)p(y)}$ gives a negative number.

But MI is always non-negative, as it can be written as $d_{KL}(p(x,y)||p(x),p(y))$, where $d_{KL}$ is the Kullback–Leibler divergence which is non-negative (the proof of the latter is not completely trivial).

Intuitively, this happens because if there is a point such that $p(x,y)<p(x)p(y)$, then the probability must "compensate" for this later, with a pair $p(x',y')>p(x')p(y')$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.