First of all, I have to mention that I am very new in the field of information theory. I have a question regarding the Shannon Entropy calculation for binary values. As far as I understood, the main assumption in the Shannon Entropy formula is that all the bits have the same weights. However, I am looking for a formula/techniques to calculate the entropy based on the weight of each bit of a binary string. Let me elaborate on this.
Suppose we have a 3-bits binary number (b2, b1, b0) and let us assume that the probability of being 0 or 1 is 0.5. Based on the Shannon's theorem the entropy considers all these bits with the same weight. If I give tell you that (b1, b0) = (00) then you can guess that the final value is either 0 or 4. But if I give you (b2,b1) = (00) you can guess that the final value is either 0 or 1. To me it means b2, b1 provides more information than knowing b1,b0. Because, you can guess the final value with less error. Is it really related to information or it is something else?
It would be good if you could provide me some links to get more information about this topic.
This is the link to my related question: Hamming Error Correction
Thanks