Conceptual overview: Self-information, Mutual information, uncertainty, entropy

Question

I would like to have a compact conceptual explanation that allows me to gain some feeling with the concepts. The following list uses $x_i, y_i$ as events of experiments $X, Y$.

• Self-information, denoted as $I(x_i)$
• Mutual information, denoted as $I(x_i; y_i)$
• Uncertainty (which is closely related to self-information and/or mutual information, I believe).
• Average mutual information, denoted as $I(X, Y)$.
• Entropy, denoted as $H(X)$, which is average self-information, I believe.

Next, there are a couple of "conditional ..." defined.

Can you sketch a compact overview of these concepts and how they relate, all intuitively?

Answer 1

Trying to answer the question myself, with my textbook at hand. So far, I am here:

• Self-information of an event: uncertainty of an event.
  $$I(x_i) = \log \frac{1}{P(x_i)} = -\log P(x_i)$$ Self information of a vector event: $$I(x_i y_i) = -\log P(x_i,~y_i)$$

• Mutual information between two events: the information one event (experiment outcome) gives us about the other event: $$I(x_i;y_i) = \log \frac{P(x_i,~y_i)}{P(x_i) P(y_i)}$$ $$I(x_i y_i) = I(x_i) + I(y_i) - I(x_i; y_i)$$

• Uncertainty: the formula of it is the same as self-information of an event. So I'll assume they are the same.

• Average mutual information: The information one gives about the other when observer the former. Clumsy notation: $$I(X; Y) = E[I(x_i; y_i)]$$

• Entropy: the average amount of information that is revealed by observing an event. (I know the relationship to Huffman coding here.) Clumsy notation:
  $$H(X) = E[I(x_i)]$$

• Conditional entropy: the amount of information (about $X$) that is left to be discovered after observing the given ($Y$). $$H(X|Y) = H(X) - I(Y;Y)$$

\begin{align} I(X; Y) &= H(X) - H(X | Y) \\ \text{mutual} &= \text{a priori} - \text{a posteriori} \end{align}

• Conditional self-information: the information that is contained within the outcome of a certain event ($x_i$), after already knowing the outcome of another event ($y_i$). $$I(x_i | y_i) = -\log P(x_i | y_i)$$ Relations: $$I(x_i | y_i) = I(x_i) - I(x_i; y_i)$$ \begin{align} I(x_i; y_i) &= I(x_i) - I(x_i | y_i) \\ \text{mutual} &= \text{a priori} - \text{a posteriori} \end{align}

Answer 2

conditional self information is the modified version of self information in which we see the amount of information carried out by X(i) in the presence of Y(i)

                      I[x(i)|y(i)]=  -log[x(i)|y(i)]