How does information theory deal with truth?
Does disinformation or lie have negative amount of bits of information?
$\begingroup$
$\endgroup$
$\begingroup$
$\endgroup$
How does information theory deal with truth?
It doesn't. It deals with messages drawn from probability distributions, not with knowledge.
Does disinformation or lie have negative amount of bits of information?
No, because that's not what information theory is about.
answered Oct 17 '16 at 15:13
-
$\begingroup$ So i wonder, if sentence "Sun is hot" had 1 bit of information, how many bits would "Sun is hot and sun is not hot" have. $\endgroup$ – mykhal Oct 17 '16 at 15:28
-
$\begingroup$ The information in a message depends only on the probability distribution from which that message was drawn, not from any meaning the message might have in English. $\endgroup$ – David Richerby Oct 17 '16 at 15:51
-
$\begingroup$ David Richerby: OK, I probably should have asked in Philosophy section :) $\endgroup$ – mykhal Oct 17 '16 at 15:56
-
3$\begingroup$ Well, the point about mathematics is that you should get the same answer whomever you ask. $\endgroup$ – David Richerby Oct 17 '16 at 16:01
$\begingroup$
$\endgroup$
Suppose we have the following deck of cards
$$\{\mbox{Ace}, 2, 3, 4, 5, 6, 7, 8\}$$
I shuffle the deck, pick a card, but do not show it to you. You are totally ignorant as to what card I picked. Hence, you model your ignorance using a uniform probability mass function over the deck. Computing the Shannon entropy, your ignorance can be measured: $3$ bits.
Suppose I send you the following message:
The card I picked is greater than $4$.
Your probability mass function is now uniform over $\{5, 6, 7, 8\}$, whose Shannon entropy is $2$ bits. By halving the set of "suspects", you gained $1$ bit of information from my message.
Suppose now that I send you the following message next:
The card I picked is a prime number.
Since $6$ and $8$ are not prime, the set of "suspects" is halved again. Your probability mass function is now uniform over $\{5,7\}$, whose Shannon entropy is $1$ bit. You gained another bit of information from my 2nd message.
Suppose I now send you a 3rd message:
Just kidding! The card I picked is not a prime number.
Your probability mass function is again uniform over $\{5, 6, 7, 8\}$. Your ignorance is again $2$ bits. In other words, you lost one bit of information when you received my 3rd message.
Perhaps you lost all trust in me. In that case, your probability mass function is back to the uniform distribution over the whole deck of cards, which corresponds to $3$ bits of ignorance.
Whenever a message makes you more "confused", whenever it widens the support of the uniform probability mass function you use to model your ignorance... well, then that message carries a negative information gain, which is another way of saying that it leads to information loss, i.e., to increased uncertainty.
answered Oct 17 '16 at 20:37
