2
$\begingroup$

If you look at page 13 of the lecture slides here there is this line

$H(Y) = H((1-\pi)(1-\alpha), \alpha, \pi(1-\alpha))$

I don't really understand what the term on right hand side is. At first I thought it's joint probability of binary random variables but that does not make sense. After a while I notice that the first parameter is the probability of $Y = 0$, the second parameter is the probability $Y = e$ and the third parameter is the probability $Y = 1$. However, I still dont really understand what this notation means, and why this equates to

$H(\alpha,(1-\alpha)) + (1-\alpha)H(\pi,(1-\pi))$

Can someone please help me out here?

Thank you!

$\endgroup$
  • 1
    $\begingroup$ What's the context? I suggest editing the question to include all relevant context -- questions are expected to be self-contained. People shouldn't have click on an external link to understand your question (and links can disappear in the future, which as it stands would make your question incomprehensible). (continued) $\endgroup$ – D.W. Feb 13 '15 at 21:53
  • $\begingroup$ Do the lecture notes define notation for expressions like $H(a,b,c)$? Could it be the entropy of a discrete random variable that takes on three values with prob. $a$,$b$,$c$ respectively? If that notation is not defined in the lecture notes, have you tried asking your instructor/TA? If they wrote the notes, they'll presumably know. $\endgroup$ – D.W. Feb 13 '15 at 21:57
4
$\begingroup$

The first thing you should realize to understand it is that the $H(Y)$ notation corresponds to the calculation of the information entropy for the states in $Y$, such that $H(Y) = -\sum\limits_{y \in Y} p(y) \log[p(y)]$. With that in mind, we can interpret the expression and equate the terms. You are correct that the three parts correspond to the probability of $0$, $e$, and $1$, where $e$ is the state of erased information. Expanding it we get:

$H((1-\pi)(1-\alpha), \alpha, \pi(1-\alpha)) $ $\quad= -(1-\pi)(1-\alpha)\log[(1-\pi)(1-\alpha)] - \alpha\log(\alpha) - \pi(1-\alpha)\log[\pi(1-\alpha)]$

This gives us the full expression for calculating the entropy of the system composed of those three states. Next, as we usually do when proving an equality, we cheat and look at the answer to see what form we need it in. Your last term has all the $\pi$ terms clumped onto one side, and all the $\alpha$ terms clumped onto another (except for where they are also a coefficient). Thus we will organize the log terms in this manner. First we use the log rules to expand the products:

$ \quad= -(1-\pi)(1-\alpha)\log(1-\pi) - (1-\pi)(1-\alpha)\log(1-\alpha) $ $\quad\phantom{=} - \alpha\log(\alpha) - \pi(1-\alpha)\log(\pi) - \pi(1-\alpha)\log(1-\alpha)$

Then we sort all the log terms in the order of the final expression:

$ \quad = -\alpha\log(\alpha) - (1-\pi)(1-\alpha)\log(1-\alpha) - \pi(1-\alpha)\log(1-\alpha) $ $ \quad\phantom{=} - \pi(1-\alpha)\log(\pi) -(1-\pi)(1-\alpha)\log(1-\pi)$

Then we simplify and factor:

$ \quad = -\alpha\log(\alpha) - (1-\alpha)\log(1-\alpha) + (1-\alpha) \cdot [ -\pi\log(\pi) - (1-\pi)\log(1-\pi) ]$

And this is equivalent to:

$$H(\alpha, 1-\alpha) + (1-\alpha) H(\pi, 1-\pi)$$

This can be interpreted as the left part being the entropy from erasing (probability $\alpha$) or not erasing, and the right part being, in the event it was not erased, the entropy from it being a 1 (probability $\pi$) or a 0. It should also make some intuitive sense that this will be equal to the entropy from the distribution of final states of $0$, $e$ (erased), and $1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.