How does information theory deal with truth?
Does disinformation or lie have negative amount of bits of information?
Computer Science Stack Exchange is a question and answer site for students, researchers and practitioners of computer science. It only takes a minute to sign up.
Sign up to join this communityHow does information theory deal with truth?
Does disinformation or lie have negative amount of bits of information?
How does information theory deal with truth?
It doesn't. It deals with messages drawn from probability distributions, not with knowledge.
Does disinformation or lie have negative amount of bits of information?
No, because that's not what information theory is about.
Suppose we have the following deck of cards
$$\{\mbox{Ace}, 2, 3, 4, 5, 6, 7, 8\}$$
I shuffle the deck, pick a card, but do not show it to you. You are totally ignorant as to what card I picked. Hence, you model your ignorance using a uniform probability mass function over the deck. Computing the Shannon entropy, your ignorance can be measured: $3$ bits.
Suppose I send you the following message:
The card I picked is greater than $4$.
Your probability mass function is now uniform over $\{5, 6, 7, 8\}$, whose Shannon entropy is $2$ bits. By halving the set of "suspects", you gained $1$ bit of information from my message.
Suppose now that I send you the following message next:
The card I picked is a prime number.
Since $6$ and $8$ are not prime, the set of "suspects" is halved again. Your probability mass function is now uniform over $\{5,7\}$, whose Shannon entropy is $1$ bit. You gained another bit of information from my 2nd message.
Suppose I now send you a 3rd message:
Just kidding! The card I picked is not a prime number.
Your probability mass function is again uniform over $\{5, 6, 7, 8\}$. Your ignorance is again $2$ bits. In other words, you lost one bit of information when you received my 3rd message.
Perhaps you lost all trust in me. In that case, your probability mass function is back to the uniform distribution over the whole deck of cards, which corresponds to $3$ bits of ignorance.
Whenever a message makes you more "confused", whenever it widens the support of the uniform probability mass function you use to model your ignorance... well, then that message carries a negative information gain, which is another way of saying that it leads to information loss, i.e., to increased uncertainty.