2
$\begingroup$

Consider a simple Bernoulli variable X

X = 1 with probability p
X = 0 with probability (1-p)

The variance is simply p(1-p). The entropy is -p*ln(p)-(1-p)*ln(1-p) Both are measures of uncertainty. What is the advantage of entropy then?

Thanks!

$\endgroup$

1 Answer 1

2
$\begingroup$

They measure different things. Given the variance, you can solve for $p$ and compute the entropy, and vice versa, but they have different interpretations and different applications. You can learn a bit about the meaning and applications of these notions in the corresponding Wikipedia articles: https://en.wikipedia.org/wiki/Entropy_(information_theory), https://en.wikipedia.org/wiki/Variance.

Analogy: if we have a sphere, I could measure its radius, or I could measure its volume. They both reflect the size of the sphere, so you could as, what is the advantage of radius, or of volume? Answer: they measure different things, so depending on what you want to do, one or the other might be more useful.

$\endgroup$
2
  • $\begingroup$ so what do they measure ?:) $\endgroup$
    – PascalIv
    Sep 25, 2023 at 13:08
  • $\begingroup$ @PascalIv, you should be able to learn more about what they measure by reading standard resources, such as the links I provided and any standard textbook. $\endgroup$
    – D.W.
    Sep 25, 2023 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.