I am learning about information theory and mutual information. However, I am quite confused with MI(Mutual information) vs. PMI(Pointwise mutual information) especially signs of MI and PMI values. Here are my questions.
Is MI values a non-negative value or it can be either positive or negative? If it is always a non-negative value, why is it ?
As I search online, the PMI can be positive or negative values and the MI is the expected value of all possible PMI. However, expected value can be positive or negative. If MI is really the expected value of PMI, why is it always positive ?
Did I misunderstand anything of MI and PMI here ? Thank you very much,