3
$\begingroup$

I am trying to fully understand the meaning behind the self-information of events. While I totally understood how to calculate it, all explanations I find online lack of an explanation why less likely events have more self-information.

I can't figure out what consequences this has and can't put it into perspective like other concepts in computer science.

What does the self-information really mean? Can someone put it into context and why exactly less likely events have a higher self-information?

Improve this question
$\endgroup$
3
$\begingroup$

The self-information is a measure of deviation from expectation of random variable in shannons (bits, the unit may vary when used in different context) when sampling a random variable.

Straight from definition it is reciprocal of probability of occurence for some event, so a little example would help.

Imagine that somewhere it rains only in one month every year (for some random number of days). From the event it is a sunny day we know almost nothing and it could be any month, so it meets the common (the most probable) outcome and we learned almost nothing, so it is not very important piece of information. But the event it rains gives us the exact month, so it is worth more in the sense that we have learned more, hence the higher self-information.

Some people call it a surprisal, which means "not the expected outcome". You need more memory to store rare event, (sorry for circular ref, but it is one of consequences, for the compression) think about the entropy encoding, it takes the most amount of bits to store the least expected events. Huffman codes and self-information.

Improve this answer
$\endgroup$
  • $\begingroup$ But how exactly we learned "more"? Is it because we simultaneously learned which months are not rainy? Does your example indicate that you want to know when it rains (meaning its your goal to find it out)? Your explanation is good, but I still miss the eureka effect to be honest. $\endgroup$ – yemerra Aug 25 '17 at 6:56
  • 1
    $\begingroup$ Well, the self-information and surprisal is the same thing. And it was defined that way, to measure the transition of "good" information. If we take events: the Sun still exist, rain, encounter with wolf, rocks starts levitation, the first one gives nothing new, rather common knowledge that the Sun is there. The rain itself is expected event, in some months more, and gives the warning to wear coat. Wolf encounter - it gives a warning, and is quite rare. The rocks levitation - is impossible. I have sorted events by rarity (I hope). $\endgroup$ – Evil Aug 25 '17 at 17:19
  • 1
    $\begingroup$ If something is common knowledge, the surprisal is 0, and we call it uninformative news. If event is common, and requires some preperation it is more informative case, with rare event (wolf) it is hard to be constantly prepared for encounter, but when it happens, it is more important to us to know in front. The levitation if rocks is impossible, so the self-information is infinite (that much it surprises us and changes that much) $\endgroup$ – Evil Aug 25 '17 at 17:20
  • 1
    $\begingroup$ Let me take wolf, why the mere rarity is surprisal and information? If it was everyday encounter, it would be of no value (people would constantly use some counter-meassures to prevent the attacks, so self-information is the measure of unpreparedness... The low expectation gives more rewards (or losses) when the event occurs. And the Shannon tried to picture a way to describe it mathematically. $\endgroup$ – Evil Aug 25 '17 at 17:26
  • 1
    $\begingroup$ Wow I think I got it now. Thank you very much for your help. $\endgroup$ – yemerra Aug 25 '17 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.