Consider a simple Bernoulli variable X

X = 1 with probability p
X = 0 with probability (1-p)

The variance is simply p(1-p). The entropy is -p*ln(p)-(1-p)*ln(1-p) Both are measures of uncertainty. What is the advantage of entropy then?



They measure different things. Given the variance, you can solve for $p$ and compute the entropy, and vice versa, but they have different interpretations and different applications. You can learn a bit about the meaning and applications of these notions in the corresponding Wikipedia articles: https://en.wikipedia.org/wiki/Entropy_(information_theory), https://en.wikipedia.org/wiki/Variance.

Analogy: if we have a sphere, I could measure its radius, or I could measure its volume. They both reflect the size of the sphere, so you could as, what is the advantage of radius, or of volume? Answer: they measure different things, so depending on what you want to do, one or the other might be more useful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.