Relation of Entropy and SNR : Based on this question and answer, I had another question that struck me and I am curious to know, if somebody can shed some light, on the following situation: $y= desired_{signal} + noise$ is transmitted where $noise$ is an Additive White Gaussian Noise. $y(t)$ is received by the receiver and the $error = y - \hat{y}$ where $\hat{y}$ is simulated at the receiver by using guessed parameters with the knowledge of the kind of model used at the transmitter.

For increasing SNR (attenuating noise) would Entropy of error decrease? In the link provided, the entropy of the transmitted signal decreases with increasing SNR since the uncertainty decreases when noise is getting attenuated. But what about the entropy of error? Should error entropy increase with increasing SNR? I am confused about this.

EDIT: The way I calculated entropy of error: Let the system model be AR(2): $y(t)= a1y(t-1) + b1y(t-2) + noise$. At the receiver end, I have $z(t) = y(t) - (a2y(t-1) + b2y(t-2))$ where $(a2,b2)$ are close guesses to ($a1,b1$). I calculated 2 entropies: $Entropy[y(t)]$ and $Entropy[z(t)]$.

z(t) will be close to zero when $a2,b2$ will be equal to $a1,b1$ in which case entropy of z(t) is found to be minimum among all different z(t) for a particular SNR value of noise. So, for a particular noise value, I have 10 pairs of (a2,b2) and for each I get 10 values of $Entropy[z]$. I chose the minimum among all $Entropy[z]$ for a particular SNR level. But, for different noise levels, as I increase the SNR at the transmitter end, the minimum entropy, $\min Entropy[z(t)]$, increases with increasing SNR. While, I found that $Entropy[y]$ decreases with increasing SNR. What should be the correct trend and why?

  • $\begingroup$ Can you explain why you think this is on-topic for computer science? This sounds like a question about statistics, or information theory / communications theory. Would this be better on the EE StackExchange? (But don't cross-post: if you want it to appear on a different SE site, flag it and ask teh moderator to migrate it.) $\endgroup$
    – D.W.
    Sep 7 '14 at 6:18
  • $\begingroup$ Entropy and information theory is a part of Computer Science and in my question I am asking about a deep intuitive relationship,if any, between noise, entropy of a signal and entropy of errors. I think this is more related to science and its application in signal processing. Without the scientific implication, it is not possible to comprehend the way entropy works. Pardon me if I am wrong. $\endgroup$
    – Ria George
    Sep 7 '14 at 18:03
  • $\begingroup$ My sense is that this kind of problem, where we're looking at signals from a contiguous domain like the real numbers and additive Gaussian noise, is more typically studied in electrical engineering, rather than computer science -- so you're more likely to reach people who can answer your question at an EE site than a CS site. (Rough rule of thumb: Information theory on continuous variables is typically studied in EE; information theory on discrete domains (e.g., a finite number of symbols) is studied in both CS and EE.) I could be wrong. Your call. It's not clearly off-topic for CS.SE. $\endgroup$
    – D.W.
    Sep 7 '14 at 19:18
  • $\begingroup$ Entropy is additive. Therefore if the entropy of the signal decreases, the entropy of the noise must increase. $\endgroup$ Sep 7 '14 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.