I would like to have a compact conceptual explanation that allows me to gain some feeling with the concepts. The following list uses $x_i, y_i$ as events of experiments $X, Y$.

  • Self-information, denoted as $I(x_i)$
  • Mutual information, denoted as $I(x_i; y_i)$
  • Uncertainty (which is closely related to self-information and/or mutual information, I believe).
  • Average mutual information, denoted as $I(X, Y)$.
  • Entropy, denoted as $H(X)$, which is average self-information, I believe.

Next, there are a couple of "conditional ..." defined.

Can you sketch a compact overview of these concepts and how they relate, all intuitively?

  • 1
    $\begingroup$ This seems to be asking for a whole textbook chapter, which I think is too broad here. However, it may have use as a reference question. Community votes, please! $\endgroup$ – Raphael Jun 9 '17 at 11:35
  • $\begingroup$ @Raphael: I am trying to answer the question myself, with my text book at hand. I think I have an overview right now. I'll change my question to make it reference style. If you are familiar within the field, can you verify my current overview? $\endgroup$ – Martijn Courteaux Jun 9 '17 at 12:01
  • $\begingroup$ Nope, others will have to do that. :) $\endgroup$ – Raphael Jun 9 '17 at 12:51

Trying to answer the question myself, with my textbook at hand. So far, I am here:

  • Self-information of an event: uncertainty of an event.
    $$I(x_i) = \log \frac{1}{P(x_i)} = -\log P(x_i)$$ Self information of a vector event: $$I(x_i y_i) = -\log P(x_i,~y_i)$$

  • Mutual information between two events: the information one event (experiment outcome) gives us about the other event: $$I(x_i;y_i) = \log \frac{P(x_i,~y_i)}{P(x_i) P(y_i)}$$ $$I(x_i y_i) = I(x_i) + I(y_i) - I(x_i; y_i)$$

  • Uncertainty: the formula of it is the same as self-information of an event. So I'll assume they are the same.

  • Average mutual information: The information one gives about the other when observer the former. Clumsy notation: $$I(X; Y) = E[I(x_i; y_i)]$$

  • Entropy: the average amount of information that is revealed by observing an event. (I know the relationship to Huffman coding here.) Clumsy notation:
    $$H(X) = E[I(x_i)]$$

  • Conditional entropy: the amount of information (about $X$) that is left to be discovered after observing the given ($Y$). $$H(X|Y) = H(X) - I(Y;Y)$$

\begin{align} I(X; Y) &= H(X) - H(X | Y) \\ \text{mutual} &= \text{a priori} - \text{a posteriori} \end{align}

  • Conditional self-information: the information that is contained within the outcome of a certain event ($x_i$), after already knowing the outcome of another event ($y_i$). $$I(x_i | y_i) = -\log P(x_i | y_i)$$ Relations: $$I(x_i | y_i) = I(x_i) - I(x_i; y_i)$$ \begin{align} I(x_i; y_i) &= I(x_i) - I(x_i | y_i) \\ \text{mutual} &= \text{a priori} - \text{a posteriori} \end{align}
  • 1
    $\begingroup$ This looks to be a long way from what you want to achieve. Note that many edits to posts are discouraged since every edit bumps the question on the homepage. Please prepare the post offline (e.g. using stackedit.io) and post it once it's ready for review. $\endgroup$ – Raphael Jun 9 '17 at 12:52

conditional self information is the modified version of self information in which we see the amount of information carried out by X(i) in the presence of Y(i)

                      I[x(i)|y(i)]=  -log[x(i)|y(i)]
  • 1
    $\begingroup$ Welcome to CS.SE! The question asks for an explanation of all of these concepts and how they relate; this answer explains only one of them. Also, it seems that the material in your answer already mostly appears in Martijn Courteaux's answer. We'd prefer that you answer questions where you can provide new information that doesn't duplicate existing answers, and that answers the complete question. $\endgroup$ – D.W. Feb 2 '18 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.