Skip to main content

Tag Info

New answers tagged probability-theory

0 votes

Expected Entropy of the Empirical Distribution

Not a closed form equation for the expected entropy, but an algorithm to compute it. Here $n$ is the number of outcomes with probabilities $[p_1, p_2, \dots, p_n]$, and $m$ is the number of samples ...
• 301
0 votes

Chernoff bound when we only have upper bound of expectation

You can obtain this statement from your given form of Chernoff's bound using a coupling. Write $\mu - \mathbf{E}[X] = i + f$ where $i$ is a nonnegative integer and $0 \le f \le 1$. Now define the ...
• 141
0 votes

Chernoff bound when we only have a lower bound of expecation

$$\Pr[X \le (1 - \delta)\alpha] \le \Pr[X \le (1 - \delta) \mathbf{E}[X]] \le \exp(-\delta^2 \mathbf{E}[X] / 2) \le \exp(-\delta^2 \alpha / 2)$$
• 141
3 votes
Accepted

Information content of a real number

In some sense I agree that $\mathcal{U}_{[0,1]}$ has "infinite information" but we should be a little more careful formalizing it. As indeed, Shannon entropy is only defined for discrete ...
1 vote

Information content of a real number

Yes. Its information content (entropy) is infinite.
• 162k

Top 50 recent answers are included