1
$\begingroup$

We know by AEP that if random variables $X_1,X_2,...$ are i.i.d. drawn from $P_X$ then the probability of the vectors in the weak typical set $$A_{\epsilon}^n = \{\vec x \in \mathcal{X}^n: |\frac{-1}{n}\log P(\vec x) - H(P_X)| \leq \epsilon \}$$ tends to 1.
I am working on a slightly different problem. Random variables $X_1,X_2,...$ are independently drawn from $P^1_X,P^2_X,..,P^n_X$ respectively. Let use denote $\tilde P_X = \frac{1}{n} \sum_{i=1}^n P^i_X$ (ie, the average of the distributions). Is it true that the sum of the vectors contained in the set $B_{\epsilon}^n$ would have a very high probability? $$B_{\epsilon}^n = \{\vec x \in \mathcal{X}^n: |\frac{-1}{n}\log P(\vec x) - H(\tilde P_X)| \leq \epsilon \}$$

$$ = \{\vec x \in \mathcal{X}^n: |\frac{-1}{n}\sum_{i=1}^n\log P^i_X(x_i) - H(\tilde P_X)| \leq \epsilon \}$$ I have no idea how to prove/disprove this. It seems like I can not use Weak law of large numbers.

$\endgroup$
1
$\begingroup$

Suppose that $X_n = n$ with probability $1$. Then $H(\tilde{P}_X) = \log n$, but the only vector with non-zero probability has $\frac{1}{n} \log \frac{1}{P(\vec{x})} = 0$.

$\endgroup$
2
  • $\begingroup$ I think the same answer is more insightful if you if define $X_n = n \mod N$ to restrict the random variables to finite symbols. $H(\tilde P_X) \approx \log N$ in that case. Thanks for the answer. $\endgroup$ Sep 21 '20 at 13:55
  • $\begingroup$ Hey, I think the statement hols if I tweak it to $\frac{-1}{n}\sum \log \tilde P_X (x_i)$ in my definition of $B_{\epsilon}^n$ $\endgroup$ Sep 23 '20 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.