2
$\begingroup$

Suppose I have a sequence of 5 IID Bernoulli variables, with probability of having value 1 being 0.5. If we observe an instance of the sequence, the information content would be 5 bits, as each variable contributes 1 bit.

But instead, if I have a single real number as a random variable, say, uniformly distributed over [0,1], would the information content of this single variable be infinite? If I write it in base 2, then this would simply be an infinite sequence of Bernoulli variables after all.

$\endgroup$
2
  • $\begingroup$ Yes, so… what is your point? $\endgroup$
    – Nathaniel
    Commented May 25 at 18:56
  • $\begingroup$ I had an informal reasoning in mind, I just wanted to make sure that I'm not missing any technical nuance. The accepted answer suits my query $\endgroup$ Commented May 26 at 17:42

2 Answers 2

3
$\begingroup$

In some sense I agree that $\mathcal{U}_{[0,1]}$ has "infinite information" but we should be a little more careful formalizing it. As indeed, Shannon entropy is only defined for discrete random variables: $$H(X) := - \sum_{x} p(x) \log p(x).$$ There are two approaches we can take to extend this to continuous distributions.

Differential entropy

What if we squint a little, and try to directly translate Shannon entropy to a continuous random variable. We could take the probability density function $f$ and propose: $$H(X) := - \int_x f(x) \log f(x).$$ Computing this for $\mathcal{U}_{[0,1]}$ we get: $$H(\mathcal{U}_{[0,1]}) := \int_{0}^{1} 1 \log 1 = 0.$$ Huh. So yeah it turns out this is called the differential entropy, and, despite having a lot of the nice mathematical properties of Shannon entropy, non-negativity is not one of them! It is generally not a meaningful notion of information content.

Limit of discrete distributions

Instead, let's define $\mathcal{D}_n$ to be the uniform distribution over the $2^n$ points $\{0, 1/2^n, ..., (2^n-1)/2^n\}$. This is equivalent to sampling $x \sim \mathcal{U}_{[0,1]}$ and then truncating after $n$ bits. Compute $$H(\mathcal{D}_n) = - \sum_{x} p(x) \log p(x) = \sum_{x} 2^{-n} \log 2^n = n.$$ As $n \to \infty$ the $\mathcal{D}_n$ converge to $\mathcal{U}_{[0,1]}$. And then $H(\mathcal{D}_n) \to \infty$ which seems to support your idea that a continuous random variable has "infinite information".

From a computer science perspective, it's more meaningful to talk about representing $x \sim \mathcal{U}_{[0,1]}$ to a certain precision, which is exactly what this setup captures. We have shown, by Shannon's source coding theorem, that any encoding scheme needs an average of $n$ bits to represent $x$ to $n$ places of binary precision.

$\endgroup$
1
$\begingroup$

Yes. Its information content (entropy) is infinite.

$\endgroup$
1
  • $\begingroup$ Specifically, if you express the number in binary, it would in general require an infinite number of bits to transmit a general real number from Alice to Bob. $\endgroup$
    – Pseudonym
    Commented May 26 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.