Suppose that I have two random processes. Process $X$ has probability 0.5 to be in state A and probability 0.5 to be in state B. Process $Y$ has probability 0.4 to be in state A and probability 0.6 to be in state B.

State B has two substates: $B_1$ and $B_2$. For both processes, the two substates are equally likely.

Now if I choose to only consider the states $A$ and $B$, then process $X$ has higher Shannon entropy. But if I consider states $A$, $B_1$ and $B_2$, then process $Y$ has higher Shannon entropy.

What is the correct way to chose the states for which you calculate the Shannon entropy?


1 Answer 1


Shannon entropy computes the average number of bits needed to represent the outcomes.

Your first probability distributions are: D1: 0.5, 0.5 and D2: 0.4, 0.6, the first of which has higher shannon entropy (maximal in fact).

In the second situation your distributions are: D3: 05, 0.25 and 0.25 and D4: 0.4, 0.3 and 0.3.

The situation is now reversed: fewer bits are needed to represent a process with outcomes following distribution D3 (0.5, 0.25 and 0.25).

The intuition is that a more optimal encoding (using bits) exploits the highest frequencies (= highest probabilities of occurrence) equipping those outcomes with shorter codes (i.e. shorter binary numbers) so that the overall average encoding cost (the average number of bits used) is lower.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.