1
$\begingroup$

Suppose $2$ bits are used to encode a message, A and B.

  • If you know $A$ is $1$, you have one bit of information.
  • If you know $A\land B$ is $1$, you have two bits of information.
  • If you know $A\land B$ is $0$, you have $0.415$ bits of information.

When is this number ever useful?

P.S: The formula for calculating the information content is $-\log_2\left(\text{probability}\right)$, and since the probability that $A\land B=0$ is $\frac{3}{4}$, we have $–\log_2\left(\frac{3}{4}\right) = 0.415$ bits.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

An amount of information does not need to be an integer, just like the weight of an object does not need to be an integer number of grams.

One might see as a source of confusion to give the same name to the unit of information and to a digit in the binary base. Of course this comes from the fact that a single equiprobable binary digit (Boolean value) conveys an amount of information equal to one.

When the distribution of a bit value is skewed, one of the amounts of information must be a fractional number because not two integer powers of $\frac12$ sum to $1$, except $\frac12+\frac12$.

In your example, the expectation of the amount of information of an and bit is

$$\frac14\text{colg}\frac14+\frac34\text{colg}\frac34=0.811\cdots$$

which shows the loss of information from the initial $2$ bits.

Improve this answer
$\endgroup$
2
  • $\begingroup$ $\text{colg}$: base-2 cologarithm. $\endgroup$
    – user16034
    May 25 at 13:03
  • $\begingroup$ Thank you! It does clearly show that A&B and A|B have less information than A or B alone. I was hoping for there to be some ingenious way of working with these log formulas, so that you could easily apply it to designing algorithms for error correcting codes and reversible circuits. I've read more about it and saw that the formulas for Mutual Information and for Conditional Entropy are useful in that situation. However, when calculating advanced algorithmic complexity, using probabilities is much more useful than using these logarithmic metrics of information content. Thank you once again! $\endgroup$
    – G S
    May 25 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.