I was following the text book by David Mackay: information theory inference and learning algorithms, this could be found online on his website.

I have question on the source coding theorem (emphasis mine):

Shannon's source coding theorem (p81): $N$ i.i.d. random variables each with entropy $H(x)$ can be compressed into more than $N \cdot H(x)$ bits with negligible risk of information loss, as $N \to \infty$; conversely if they are compressed into fewer than $N \cdot H(x)$ bits it is virtually certain that information will be lost.

My question is the emphasised part of the above, what happens when you go below $N \cdot H(x) bits?

There is an example in the textbook (p77), which he shows that, as $N$ increases, the average entropy per symbol approaches $H(X)$, for the risk tolerance between $0< \delta< 1$. (So it becomes more flat as indicate on the diagram).

Therefore, $N \cdot H(x)$ is the number of bits you can compress no matter how much risk tolerance you are willing to take. In the theorem above, however, it is saying that information will be lost if you go below $N \cdot H(X)$. My question is, how much information will be lost?

I was following the text book by David Mackay: information theory inference and learning algorithms, this could be found online on his website.

I have question on the source coding theorem (emphasis mine):

Shannon's source coding theorem (p81): $N$ i.i.d. random variables each with entropy $H(x)$ can be compressed into more than $N \cdot H(x)$ bits with negligible risk of information loss, as $N \to \infty$; conversely if they are compressed into fewer than $N \cdot H(x)$ bits it is virtually certain that information will be lost.

My question is the emphasised part of the above, what happens when you go below $N \cdot H(x)$ bits?

There is an example in the textbook (p77), which he shows that, as $N$ increases, the average entropy per symbol approaches $H(X)$, for the risk tolerance between $0< \delta< 1$. (So it becomes more flat as indicate on the diagram).

Therefore, $N \cdot H(x)$ is the number of bits you can compress no matter how much risk tolerance you are willing to take. In the theorem above, however, it is saying that information will be lost if you go below $N \cdot H(X)$. My question is, how much information will be lost?

I was following the text book by David Mackay: information theory inference and learning algorithms, this could be found online on his website:http://www.cs.toronto.edu/~mackay/itprnn/book.pdf.

I have question on the source coding theorem: Shannon's source coding theorem (p81): N i.i.d. random variables each with entropy H(x) can be compressed into more than NH(x) bits with negligible risk of information loss, as N->\infinity; **conversely if they are compressed into fewer than NH(x) bits it is virtually certain that information will be lost.**

My question is the bold part of the above, what happen when you go below N*H(x) bits?

There is an example in the textbook (p77), which he shows that, as N increase, the average entropy per symbol approaches to H(X), for the risk tolerance between 0< delta< 1. (So it becomes more flat as indicate on the diagram). Therefore, NH(x) is the number of bits you can compress no matter how much risk tolerance you are willing to take. In the definition above, however, it is saying that information will be lost if you go below NH(X). My question is, how much information will be lost?

I was following the text book by David Mackay: information theory inference and learning algorithms, this could be found online on his website.

I have question on the source coding theorem (emphasis mine):

Shannon's source coding theorem (p81): $N$ i.i.d. random variables each with entropy $H(x)$ can be compressed into more than $N \cdot H(x)$ bits with negligible risk of information loss, as $N \to \infty$; conversely if they are compressed into fewer than $N \cdot H(x)$ bits it is virtually certain that information will be lost.

My question is the emphasised part of the above, what happens when you go below $N \cdot H(x) bits?

There is an example in the textbook (p77), which he shows that, as $N$ increases, the average entropy per symbol approaches $H(X)$, for the risk tolerance between $0< \delta< 1$. (So it becomes more flat as indicate on the diagram).

Therefore, $N \cdot H(x)$ is the number of bits you can compress no matter how much risk tolerance you are willing to take. In the theorem above, however, it is saying that information will be lost if you go below $N \cdot H(X)$. My question is, how much information will be lost?

I was following the text book by David Mackay: information theory inference and learning algorithms, this could be found online on his website:http://www.cs.toronto.edu/~mackay/itprnn/book.pdf.

I have question on the source coding theorem.

My question is, what happen when you go below N*H(x) bits?

There is an example in the textbook (p77), which he shows that, as N increase, the average H(X^N) per symbol approaches to H(X), for the risk tolerance between 0< delta< 1. (So it becomes more flat as indicate on the diagram). Therefore, NH(x) is the number of bits you can compress no matter how much risk tolerance you are willing to take. In the definition above, however, it is saying that information will be lost if you go below NH(X). My question is, how much information will be lost?

I was following the text book by David Mackay: information theory inference and learning algorithms, this could be found online on his website:http://www.cs.toronto.edu/~mackay/itprnn/book.pdf.

I have question on the source coding theorem: Shannon's source coding theorem (p81): N i.i.d. random variables each with entropy H(x) can be compressed into more than NH(x) bits with negligible risk of information loss, as N->\infinity; **conversely if they are compressed into fewer than NH(x) bits it is virtually certain that information will be lost.**

My question is the bold part of the above, what happen when you go below N*H(x) bits?

There is an example in the textbook (p77), which he shows that, as N increase, the average entropy per symbol approaches to H(X), for the risk tolerance between 0< delta< 1. (So it becomes more flat as indicate on the diagram). Therefore, NH(x) is the number of bits you can compress no matter how much risk tolerance you are willing to take. In the definition above, however, it is saying that information will be lost if you go below NH(X). My question is, how much information will be lost?