2
$\begingroup$

I was self learning about entropy and came across this equation. $$ H = - \sum p(x) \log p(x) $$

The equation for entropy in expected value, $$ H(x) = \operatorname*{\mathbb{E}}_{X \sim P}[I(x)] = -\operatorname*{\mathbb{E}}_{X \sim P}[\log P(x)]. $$

But the expected value is written as

$$ \mathbb{E}[X] = \sum_{i=1}^k x_i p_i = x_1p_1 + x_2p_2 + \cdots + x_k p_k $$

Using the above expected value formula, I expected the entropy equation looks something like this

$$H(x)= -\operatorname*{\mathbb{E}}_{x \sim P(x)}[\log P(x)]= - \sum xP(x)\log P(x) $$

where is the $x$ gone in the real entropy formula in summation notation?

$\endgroup$
2
  • $\begingroup$ In your last equation, the equation that you think it should look like, what is $x$? $\endgroup$
    – Pseudonym
    Commented Jun 19, 2020 at 3:37
  • $\begingroup$ I am not mathematically skilled and I might be wrong. I took the $x$ from expected value formula $E[X]$ $\endgroup$
    – Eka
    Commented Jun 19, 2020 at 14:22

1 Answer 1

2
$\begingroup$

Here is the definition of the expectation of a discrete random variable $Y$: $$ \mathbb{E}[Y] = -\sum_y \Pr[Y = y] \cdot y. $$ In your case, $Y = \log P(X)$, where $X \sim P$. Therefore $$ \mathbb{E}[X] = \sum_y \Pr[\log P(X) = y] \cdot y. $$ Notice that $$ \Pr[-\log P(X) = y] = \sum_{x\colon \log P(x)=y} \Pr[X = x] \cdot y = \sum_{x\colon \log P(x)=y} \Pr[X = x] \cdot \log P(x). $$ Therefore $$ \mathbb{E}[X] = \sum_y \sum_{x\colon \log P(x) = y} \Pr[X = x] \cdot \log P(x) = \sum_x \Pr[X = x] \log P(x) = \sum_x P(x) \log P(x). $$

$\endgroup$
4
  • $\begingroup$ Why $Y=\log P(X)$ is a random variable? $P(X)$ is not a random variable (stats.stackexchange.com/questions/300187/…). $\endgroup$
    – ado sar
    Commented Dec 3, 2022 at 14:05
  • $\begingroup$ If you apply a (measurable) function to a random variable, you get another random variable. $\endgroup$ Commented Dec 4, 2022 at 17:04
  • $\begingroup$ So can I think it like a composition $Y=f(g(X))$ where $f= \log$ and $g=P$? $\endgroup$
    – ado sar
    Commented Dec 4, 2022 at 21:23
  • $\begingroup$ If you want. You can also think of $\log P$ as a single function. $\endgroup$ Commented Dec 5, 2022 at 5:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.