Information content of a real number

Suppose I have a sequence of 5 IID Bernoulli variables, with probability of having value 1 being 0.5. If we observe an instance of the sequence, the information content would be 5 bits, as each variable contributes 1 bit.

But instead, if I have a single real number as a random variable, say, uniformly distributed over [0,1], would the information content of this single variable be infinite? If I write it in base 2, then this would simply be an infinite sequence of Bernoulli variables after all.

• Yes, so… what is your point? Commented May 25 at 18:56
• I had an informal reasoning in mind, I just wanted to make sure that I'm not missing any technical nuance. The accepted answer suits my query Commented May 26 at 17:42

In some sense I agree that $$\mathcal{U}_{[0,1]}$$ has "infinite information" but we should be a little more careful formalizing it. As indeed, Shannon entropy is only defined for discrete random variables: $$H(X) := - \sum_{x} p(x) \log p(x).$$ There are two approaches we can take to extend this to continuous distributions.

Differential entropy

What if we squint a little, and try to directly translate Shannon entropy to a continuous random variable. We could take the probability density function $$f$$ and propose: $$H(X) := - \int_x f(x) \log f(x).$$ Computing this for $$\mathcal{U}_{[0,1]}$$ we get: $$H(\mathcal{U}_{[0,1]}) := \int_{0}^{1} 1 \log 1 = 0.$$ Huh. So yeah it turns out this is called the differential entropy, and, despite having a lot of the nice mathematical properties of Shannon entropy, non-negativity is not one of them! It is generally not a meaningful notion of information content.

Limit of discrete distributions

Instead, let's define $$\mathcal{D}_n$$ to be the uniform distribution over the $$2^n$$ points $$\{0, 1/2^n, ..., (2^n-1)/2^n\}$$. This is equivalent to sampling $$x \sim \mathcal{U}_{[0,1]}$$ and then truncating after $$n$$ bits. Compute $$H(\mathcal{D}_n) = - \sum_{x} p(x) \log p(x) = \sum_{x} 2^{-n} \log 2^n = n.$$ As $$n \to \infty$$ the $$\mathcal{D}_n$$ converge to $$\mathcal{U}_{[0,1]}$$. And then $$H(\mathcal{D}_n) \to \infty$$ which seems to support your idea that a continuous random variable has "infinite information".

From a computer science perspective, it's more meaningful to talk about representing $$x \sim \mathcal{U}_{[0,1]}$$ to a certain precision, which is exactly what this setup captures. We have shown, by Shannon's source coding theorem, that any encoding scheme needs an average of $$n$$ bits to represent $$x$$ to $$n$$ places of binary precision.

Yes. Its information content (entropy) is infinite.

• Specifically, if you express the number in binary, it would in general require an infinite number of bits to transmit a general real number from Alice to Bob. Commented May 26 at 10:53