# How is expected value in entropy derived?

I was self learning about entropy and came across this equation. $$H = - \sum p(x) \log p(x)$$

The equation for entropy in expected value, $$H(x) = \operatorname*{\mathbb{E}}_{X \sim P}[I(x)] = -\operatorname*{\mathbb{E}}_{X \sim P}[\log P(x)].$$

But the expected value is written as

$$\mathbb{E}[X] = \sum_{i=1}^k x_i p_i = x_1p_1 + x_2p_2 + \cdots + x_k p_k$$

Using the above expected value formula, I expected the entropy equation looks something like this

$$H(x)= -\operatorname*{\mathbb{E}}_{x \sim P(x)}[\log P(x)]= - \sum xP(x)\log P(x)$$

where is the $$x$$ gone in the real entropy formula in summation notation?

• In your last equation, the equation that you think it should look like, what is $x$? Commented Jun 19, 2020 at 3:37
• I am not mathematically skilled and I might be wrong. I took the $x$ from expected value formula $E[X]$
– Eka
Commented Jun 19, 2020 at 14:22

Here is the definition of the expectation of a discrete random variable $$Y$$: $$\mathbb{E}[Y] = -\sum_y \Pr[Y = y] \cdot y.$$ In your case, $$Y = \log P(X)$$, where $$X \sim P$$. Therefore $$\mathbb{E}[X] = \sum_y \Pr[\log P(X) = y] \cdot y.$$ Notice that $$\Pr[-\log P(X) = y] = \sum_{x\colon \log P(x)=y} \Pr[X = x] \cdot y = \sum_{x\colon \log P(x)=y} \Pr[X = x] \cdot \log P(x).$$ Therefore $$\mathbb{E}[X] = \sum_y \sum_{x\colon \log P(x) = y} \Pr[X = x] \cdot \log P(x) = \sum_x \Pr[X = x] \log P(x) = \sum_x P(x) \log P(x).$$
• Why $Y=\log P(X)$ is a random variable? $P(X)$ is not a random variable (stats.stackexchange.com/questions/300187/…). Commented Dec 3, 2022 at 14:05
• So can I think it like a composition $Y=f(g(X))$ where $f= \log$ and $g=P$? Commented Dec 4, 2022 at 21:23
• If you want. You can also think of $\log P$ as a single function. Commented Dec 5, 2022 at 5:45