1
$\begingroup$

First of all, I have to mention that I am very new in the field of information theory. I have a question regarding the Shannon Entropy calculation for binary values. As far as I understood, the main assumption in the Shannon Entropy formula is that all the bits have the same weights. However, I am looking for a formula/techniques to calculate the entropy based on the weight of each bit of a binary string. Let me elaborate on this.

Suppose we have a 3-bits binary number (b2, b1, b0) and let us assume that the probability of being 0 or 1 is 0.5. Based on the Shannon's theorem the entropy considers all these bits with the same weight. If I give tell you that (b1, b0) = (00) then you can guess that the final value is either 0 or 4. But if I give you (b2,b1) = (00) you can guess that the final value is either 0 or 1. To me it means b2, b1 provides more information than knowing b1,b0. Because, you can guess the final value with less error. Is it really related to information or it is something else?

It would be good if you could provide me some links to get more information about this topic.

This is the link to my related question: Hamming Error Correction

Thanks

| cite | improve this question | |
$\endgroup$
3
$\begingroup$

Take a look at rate-distortion theory, which studies information transmission in settings where a certain distortion is allowed (such as audio, image, or video compression).

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I may be misunderstanding your question but I think you may be misinterpreting error. Measuring error should not be based upon the distance between the values, rather the number of possible discrete values given a particular state. That is, the fact that the distance between messages (4) and (0) is greater than (1) and (0) is inconsequential.

For either of your scenarios, a given partial message (ie. x00 or 00x) has at most 2 possible messages. Given that all bits have the same state probabilities, the probability of guessing wrong final value is the same in either scenario. So, knowing a particular two bit states does not provide more (or less) information about the third.

All bits of an encoded message do not have to have the same probability. I do not see the immediate use case for associating a weight with each bit. Could you elaborate on what you are trying to accomplish? To me it sounds like you want to artificially bias the entropy based upon the state of each bit, but I can't see why that couldn't be accomplished by simply changing the probability associated with each bit state and using the standard entropy equations found at: https://en.wikipedia.org/?title=Information_theory

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I totally agree knowing a particular two bit states does not provide more information about the third. But isn't true to say that knowing two last bit provide more information about the final value? My main goal here is to find the best rate such that the decoded code is close enough to the encoded code. Another word to say is what is the best possible achievable rate if we relax the strict constraint about the error. In my application, I don't care about the existence of the error but I care about the absolute difference between encoded message and decoded message. $\endgroup$ – AmirC Jun 25 '15 at 20:30
  • $\begingroup$ I think of the encoding as a path through a binary tree. Since all edges of the tree has the same weight (0.5), knowing any two bits of three will narrow down the possible final value to two values. So, no I don't agree with that the last bits provide more information about the final value. $\endgroup$ – putnampp Jun 25 '15 at 20:53
  • $\begingroup$ I understand what you are saying. Maybe I am not using the right term for this problem. It is true that knowing any two bits of three will bring the uncertainty to the same level. However, knowing the position of these two bits reduce the distortion to two different level. So if I understand correctly, information concerns about the uncertainty and not distortion. I think there should be a term that considers both uncertainty and distortion together. $\endgroup$ – AmirC Jun 26 '15 at 0:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.