I was self learning about entropy and came across this equation. $$ H = - \sum p(x) \log p(x) $$

The equation for entropy in expected value, $$ H(x) = \operatorname*{\mathbb{E}}_{X \sim P}[I(x)] = -\operatorname*{\mathbb{E}}_{X \sim P}[\log P(x)]. $$

But the expected value is written as

$$ \mathbb{E}[X] = \sum_{i=1}^k x_i p_i = x_1p_1 + x_2p_2 + \cdots + x_k p_k $$

Using the above expected value formula, I expected the entropy equation looks something like this

$$H(x)= -\operatorname*{\mathbb{E}}_{x \sim P(x)}[\log P(x)]= - \sum xP(x)\log P(x) $$

where is the $x$ gone in the real entropy formula in summation notation?

  • $\begingroup$ In your last equation, the equation that you think it should look like, what is $x$? $\endgroup$ – Pseudonym Jun 19 '20 at 3:37
  • $\begingroup$ I am not mathematically skilled and I might be wrong. I took the $x$ from expected value formula $E[X]$ $\endgroup$ – Eka Jun 19 '20 at 14:22

Here is the definition of the expectation of a discrete random variable $Y$: $$ \mathbb{E}[Y] = -\sum_y \Pr[Y = y] \cdot y. $$ In your case, $Y = \log P(X)$, where $X \sim P$. Therefore $$ \mathbb{E}[X] = \sum_y \Pr[\log P(X) = y] \cdot y. $$ Notice that $$ \Pr[-\log P(X) = y] = \sum_{x\colon \log P(x)=y} \Pr[X = x] \cdot y = \sum_{x\colon \log P(x)=y} \Pr[X = x] \cdot \log P(x). $$ Therefore $$ \mathbb{E}[X] = \sum_y \sum_{x\colon \log P(x) = y} \Pr[X = x] \cdot \log P(x) = \sum_x \Pr[X = x] \log P(x) = \sum_x P(x) \log P(x). $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.