5
$\begingroup$

I want to calculate the Entropy of the phrase "Eile mit Weile". I found the probability of each letter as the following

$$P(e)=\frac{4}{12}$$ $$P(i)=\frac{3}{12}$$ $$P(l)=\frac{2}{12}$$ $$P(m)=\frac{1}{12}$$ $$P(t)=\frac{1}{12}$$ $$P(w)=\frac{1}{12}$$

Then I used the formula for entropy $$H(p_1,...,p_k)=-\sum_{i=1}^{k}p_i\log_2(p_i)$$ $$H(p_1,...,p_k)=-(\frac{4}{12}\log_2(\frac{4}{12})+\frac{3}{12}\log_2(\frac{3}{12})+\frac{2}{12}\log_2(\frac{2}{12})+\frac{1}{12}\log_2(\frac{1}{12})+\frac{1}{12}\log_2(\frac{1}{12})+\frac{1}{12}\log_2(\frac{1}{12}))≈2.355388542 \ Bits$$ But the correct answer is $1.63263$ Bits. I've made a mistake somewhere but can't figure out what I did wrong.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Keep in mind that this depends on many assumptions you implicitly took. For instance, you disregard spaces, and the case of each letter. You also assume that each letter is independent, and is generated according to the frequencies of this particular sentence. With the same method, the entropy of "A AaA aa" would be zero, which could be absurd in several scenarios. $\endgroup$ – chi Apr 2 '18 at 16:22
10
$\begingroup$

The two answers agree, with the following change: it's 1.63263 nats, not bits. That is, the value 1.63263 is calculated using the natural logarithm.

Improve this answer
$\endgroup$
7
  • $\begingroup$ How do I convert from bits to nats? And also, would I have to do this conversion every time I calulated the entropy? $\endgroup$ – Ski Mask Apr 2 '18 at 12:49
  • $\begingroup$ It's a good exercise in logarithms to figure out how to convert from bits to nats and vice versa. $\endgroup$ – Yuval Filmus Apr 2 '18 at 12:50
  • $\begingroup$ Whether you have to do it or not, it's up to you. You could do the calculation using the natural logarithm in the first place if you want a result in nats. $\endgroup$ – Yuval Filmus Apr 2 '18 at 12:50
  • $\begingroup$ $1$ nat = $\frac{1}{In{2}}$bits and I want to find out what $2.35538854$bits is equal to, I'd just do cross multiplication but I come to the answer $28.2646625$nats - which is wrong $\endgroup$ – Ski Mask Apr 2 '18 at 13:01
  • $\begingroup$ When I did the calculation it worked out. Try again until it works out for you as well. $\endgroup$ – Yuval Filmus Apr 2 '18 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.