Conceptual question about entropy and information - Computer Science Stack Exchange most recent 30 from cs.stackexchange.com 2019-09-20T03:46:59Z https://cs.stackexchange.com/feeds/question/28756 https://creativecommons.org/licenses/by-sa/4.0/rdf https://cs.stackexchange.com/q/28756 4 Conceptual question about entropy and information Ria George https://cs.stackexchange.com/users/20317 2014-07-28T02:16:25Z 2014-07-28T13:04:38Z <p>Shannon's entropy measures the information content by means of probability. Is it the information content or the information that increases or decreases with entropy? Increase in entropy means that we are more uncertain about what will happen next. </p> <ol> <li><p>What I would like to know is if entropy increases, does this mean that information increases? </p></li> <li><p>If there are 2 signals, one is the desired and the other is the measurement signal. Let error be the difference between the two. Or error can be the estimation error in the context of weight learning.</p></li> </ol> <p>What can we infer if the entropy of this error term decreases? Can we conclude that the error is reducing and the system is behaving close to the desired signal's behavior?</p> <p>Shall be grateful for these clarifications</p> https://cs.stackexchange.com/questions/28756/-/28760#28760 4 Answer by Wandering Logic for Conceptual question about entropy and information Wandering Logic https://cs.stackexchange.com/users/7459 2014-07-28T12:51:08Z 2014-07-28T12:51:08Z <p>Information = Entropy = Surprise = Uncertainty = How Much You Learn By Making an Observation. They all increase or decrease together. The <em>entropy</em> of a random variable $X$ is just another number summarizing some quality of that random variable. Just like the mean of a random variable is the expected value of $X$ or the variance of a random variable is the expected value of $(X - \mu)^2$, the entropy is just the expected value of some function, $f(X)$ of the random variable $X$. You find expectations of functions by using $\mathbb{E}[f(X)] = \sum_{x\in X}p(x)f(x).$ In this case the function of $X$ you care about is the log (base 2) of the probability mass function.</p> <p>$$H(X) = \mathbb{E}[-\log_2 P(X)] = -\sum_{x \in X} p(x) \log_2 p(x).$$</p> <p>This particular expectation is useful because it doesn't depend on the actual values that $X$ can take on, just the probabilities of those values. So you can use it to talk about situations where you aren't sending numbers, or where the numbers are just arbitrarily assigned to particular messages or symbols that you need to send.</p> <p>What you want for your second question is the <a href="https://en.wikipedia.org/wiki/Information_theory#Conditional_entropy_.28equivocation.29" rel="nofollow"><em>conditional entropy</em></a> of the measurement random variable $X$ given the random variable $Y$ that represents what was sent. When there is no error the conditional entropy will be 0, when there is error the conditional entropy will be greater than 0.</p>