What units should Shannon entropy be measured in?

The only examples I've seen use bits as a measurement of entropy, but all these examples happen to use binary code alphabets. If we wanted to see how well a coding with a code alphabet of length n works, would we measure entropy in units of n?

Or would it make sense to stay using bits if we're comparing codings with binary and n-length code alphabets?

• The formulas involving entropy use the logarithm function. Since the base of the logarithms is usually taken to be $2$, entropy is measured in bits. If natural logarithms are used, the entropy is measured in nats. Since $$\log_a(x) = (\log_a(b))\log_b(x)$$ where $\log_a(b)$ is a constant for any given $a$ and $b$, the choice of logarithm base merely scales the entropy values by a constant factor, and comparisons between entropy values give the same result regardless of which units are used to express both entropies. – Dilip Sarwate Oct 22 '12 at 22:23
• Please post answers as answers, not as comments. – David Richerby Nov 17 '14 at 13:46

1 Answer

Generally bit was introduced by Shannon in 1948. Using bit instead of any other information measurement unit it's because of convenience any information can be represented by single bit or by set of bits.

I bet you already ran into most popular example which is coin tossing. You can represent head as 1 and tails as 0. You can also use stream of bits as representing more complex information as integer.

Simple example

Consider W as information about current week day you might use 7bit length stream as representation of week. Eg Monday would be represented as 1000000 and Wednesday as 0010000. So each bit in stream represents one day.

You should always consider domain for your problem and use bit encoding which fits best. Using code alphabet of length n shows universality of that particular encoding (like ASCII or Unicode).

Another thing is that bit can be efficently represented in low powered (5v) circuits. There exists interference detection and fixing functions for such circuits.

Consider domain different then bit, eg real numbers, and let them represent information using some relation R. Now because real numbers are dense even minimal interference will have impact on end result.