3
$\begingroup$

I got a bit of how Shannon explained to find the number of bits required to represent a message and Shannon's Entropy.But it's natural to know that to code alphabet letters you need 5 bits of information starting from 0000 to 11001.

Is that because at his time no one knew how to represent bits in information that Shannon came with looking at the unpredictability of the message to determine the number of bits needed to represent a message?

It's really getting confusing why would one need to consider terms like entropy,probability etc as said in Coding Theory to get the number of bits to represent a message when we already know that to represent 32 letters we need 5 bits.

Is there really any other reason behind why did Shannon came with these terms in the field of communication other than finding the number of bits required to represent a message?

$\endgroup$
5
$\begingroup$

Yes, of course. Shannon's work was on the transmission of information over discrete, noisy channels. It centred on the trade-off between bandwidth and signal-to-noise ratio, and, grossly over-simplifying it, in particular what the maximum transmission rates could be with and without the presence of noise induced error and how to construct suitable codes. This involves coding theory1 and probability theory, which is where some of things that have you confused come from.

It is also incorrect that you need 5 bits to encode letters. If we have a known frequency distribution of the letters, we can encode the more frequent ones with fewer bits, and less frequent ones with more, which will give much shorter messages that the simple 5-bits-for-everything coding. For example, the entropy given by the usual distribution of English is 4.219 bits per symbol, so we can encode common symbols like 'e' with perhaps 1 bit, and uncommon symbols like 'q' with 5 or 6 bits, an on average we use less than 5 bits per symbol. If we use more cunning coding schemes, for example ones that take into account digram and trigram frequencies, we can get this average down to a little over 2 bits per symbol by using short codes for common two and three symbol groups.

Also, notably, your title is backwards, Shannon's Entropy is a measure of information, not the other way around. It is not the only one possible, however Shannon argues that it is the most natural and intuitive.

The full extent of Shannon's (seminal) work on information theory is too large to encompass in this format, but there is plenty of further reading. The wikipedia entry on information theory is reasonably approachable, and the page on coding theory also adds something. For the real stuff of course, you can just read the paper in which Shannon invented information theory essentially out of whole cloth; "A Mathematical Theory of Communication"2.

Footnotes

  1. This is not not suggest that Shannon was applying coding theory, he was inventing new ideas that would become large parts of coding theory.
  2. Claude E. Shannon, "A Mathematical Theory of Communication", The Bell System Technical Journal, 27:379-423,623-656, 1948.
$\endgroup$
  • $\begingroup$ :I still can't get the benefit in coding less bits with popular letters like e and giving more bits to rare letters like z.Could you explain why do we have to give less bits to e.Why we couldn't give less bits to z too to make both the less and popular letters bear a minimum cost.Isn't q sufficient with 5 bits should we go for 6 bits? $\endgroup$ – justin Mar 12 '15 at 4:36
  • $\begingroup$ @justin it's a trade off, because you expect to see e a lot more than q or z, you can code the message as a whole in (probably) fewer bits if you don't have to use as much space for e, to make it work, you have to let some symbols take up more bits. A Huffman Code is one example of this in practice (particularly see the box-out example on the side, the coding reduces the message length from 180 bits (5 bits per symbol) to 135 bits (variable length encoding)). $\endgroup$ – Luke Mathieson Mar 12 '15 at 5:11
  • $\begingroup$ :Okay.Could you tell why does the information decrease when the popularity of a letter increases.Is it because it is natural to happen like that or are we actually giving a less weight(less bit of information)to the popular letter to bear a minimum cost to send the message. $\endgroup$ – justin Mar 12 '15 at 5:58
  • $\begingroup$ @justin The information content of something that we expect is lower. Consider the (semi-)hypothetical situation where we know exactly what character comes next in the message - we wouldn't need to code or transmit this message this at all, we could just write it at the receiving end, so it needs zero bits (you can extend this to any length message whose probability is 1). Now consider something that has very high probability to occur, the information (remember we're using the information theory definition of information here) of this high probability event is very low - we expect it to... $\endgroup$ – Luke Mathieson Mar 12 '15 at 6:03
  • $\begingroup$ ... happen (a lot) - so we can encode it using fewer bits. This ends up saving us in the long run, so the practical application of this is that we can encode things this way, and we'll save bits. $\endgroup$ – Luke Mathieson Mar 12 '15 at 6:04
1
$\begingroup$

There's a flaw in your premise. Information isn't said to be a measure of Shannon's entropy (you've got it backwards). Rather, Shannon's entropy is a measure of information. See https://en.wikipedia.org/wiki/Entropy_%28information_theory%29.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.