I want to calculate the Entropy of the phrase "Eile mit Weile". I found the probability of each letter as the following

$$P(e)=\frac{4}{12}$$ $$P(i)=\frac{3}{12}$$ $$P(l)=\frac{2}{12}$$ $$P(m)=\frac{1}{12}$$ $$P(t)=\frac{1}{12}$$ $$P(w)=\frac{1}{12}$$

Then I used the formula for entropy $$H(p_1,...,p_k)=-\sum_{i=1}^{k}p_i\log_2(p_i)$$ $$H(p_1,...,p_k)=-(\frac{4}{12}\log_2(\frac{4}{12})+\frac{3}{12}\log_2(\frac{3}{12})+\frac{2}{12}\log_2(\frac{2}{12})+\frac{1}{12}\log_2(\frac{1}{12})+\frac{1}{12}\log_2(\frac{1}{12})+\frac{1}{12}\log_2(\frac{1}{12}))≈2.355388542 \ Bits$$ But the correct answer is $1.63263$ Bits. I've made a mistake somewhere but can't figure out what I did wrong.

  • 2
    $\begingroup$ Keep in mind that this depends on many assumptions you implicitly took. For instance, you disregard spaces, and the case of each letter. You also assume that each letter is independent, and is generated according to the frequencies of this particular sentence. With the same method, the entropy of "A AaA aa" would be zero, which could be absurd in several scenarios. $\endgroup$
    – chi
    Commented Apr 2, 2018 at 16:22

1 Answer 1


The two answers agree, with the following change: it's 1.63263 nats, not bits. That is, the value 1.63263 is calculated using the natural logarithm.

  • $\begingroup$ How do I convert from bits to nats? And also, would I have to do this conversion every time I calulated the entropy? $\endgroup$
    – Ski Mask
    Commented Apr 2, 2018 at 12:49
  • $\begingroup$ It's a good exercise in logarithms to figure out how to convert from bits to nats and vice versa. $\endgroup$ Commented Apr 2, 2018 at 12:50
  • $\begingroup$ Whether you have to do it or not, it's up to you. You could do the calculation using the natural logarithm in the first place if you want a result in nats. $\endgroup$ Commented Apr 2, 2018 at 12:50
  • $\begingroup$ $1$ nat = $\frac{1}{In{2}}$bits and I want to find out what $2.35538854$bits is equal to, I'd just do cross multiplication but I come to the answer $28.2646625$nats - which is wrong $\endgroup$
    – Ski Mask
    Commented Apr 2, 2018 at 13:01
  • $\begingroup$ When I did the calculation it worked out. Try again until it works out for you as well. $\endgroup$ Commented Apr 2, 2018 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.