I am solving a problem of information theory. The problem reads,

Consider a stationary memoryless channel specified by the channel matrix $T = \begin{pmatrix}1-q&q\\r&1-r\end{pmatrix}$. Let the random variables $X$ and $Y$ be the input and the output of T, respectively. Let the input alphabet of $T$ be $\left\{0,1\right\}$ and $Pr(X=0)$ = $p$. Let us denote the conditional entropy $H(Y|X)$ and the mutual information $I(X;Y)$ as $f(p)$ and $g(p)$,respectively.

Question. Compare the following four quantities $Q_{1},Q_{2},Q_{3},Q_{4}$

  • $Q_{1} = f(\alpha u + (1-\alpha)v)$,
  • $Q_{2} = \alpha f(u)+(1-\alpha)f(v)$,
  • $Q_{3} = g(\alpha u + (1-\alpha)v)$,
  • $Q_{4} = \alpha g(u)+(1-\alpha)g(v)$

where $u,v$ and $\alpha$ satisfy $0<u<v<1$ and $0<\alpha<1$.

I have gotten $\begin{align} f(p) &= -pq\log q -p(1-q)\log(1-q) -(1-p)r\log r-(1-p)(q-r)\log(1-r)\\ g(p) &= -\left(p\left(1-q\right)+\left(1-p\right)r\right)\log\left(p\left(1-q\right)+\left(1-p\right)r\right) -\left(pq+\left(1-p\right)\left(1-r\right)\right)\log\left(pq+\left(1-p\right)\left(1-r\right)\right) - f(p) \end{align}$

using the definitions of conditional entropy and mutual information. Also, by differentiating $g(p)$, I confirmed that g(p) is concave. Therefore, I concluded that $Q_{3} \ge Q_{4}$. Moreover, $Q_{1}=Q_{2}$ since $f(p)$ is a linear function concerning $p$. I cannot get other relations as I thought $g(p)$ can be larger or smaller than $f(p)$ depending on $p$. Can anyone give me any hint?

  • 1
    $\begingroup$ Have you tried plugging in some example values for $p,q,r$ to see what happens in a variety of concrete cases? I suspect that might give you some hints about what relationships could possibly be true. $\endgroup$ – D.W. Jul 16 '14 at 16:53
  • $\begingroup$ Thank you for your reply. I plotted graphs for pairs of $(p,q,r)$ and found out that when $q + r$ is fairly large or small, $g(p) - f(p) > 0$ most of the time and when $q$ or $r$ is close to 0.5, $g(p) - f(p) < 0$ most of the time. This means that when the channel randomly output a value(i.e. error rate is close to 0.5), there is very little information the receiver can obtain, therefore $g(p)$ becomes pretty small(or becomes even zero). $\endgroup$ – hitochan Jul 17 '14 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.