I have a totally random source of signal data that looks like a typical normal distribution. I've included an image as I like pictures:-


The source has a mean of 0, and a standard deviation of 1. It's analogue and therefore of infinite resolution. Very typical.

I then sample the signal with a Tricorder and record the raw data. Clearly as the original signal is random, irreducible information entropy is accrued at some rate, and will be smaller (in bits) than the raw data. My particular Tricorder model has a sample resolution setting, so for example I can record at anything from 1 bit resolution /sample to say 48 bits /sample. You might think of this as a quantization setting.

The crux of my question is: what is the relationship between the recorded entropy rate and the level of quantization? I'm hoping for either a formula or an example calculation such as 4.5 bits /sample with 10 bit quantization.

There is (perhaps) a similar question at Compressing normally distributed data, but I'm not sure and it doesn't really deal with sample bit depth in the same way. I'm developing the argument that real world entropy is generated by the observer, not the underlying process.

PS. Read analogue to digital converter for Tricorder.

  • $\begingroup$ What do you mean by "recorded entropy rate"? Do you mean "the entropy of the recorded (quantized) value"? One tricky aspect here is: when you say "8 bits/sample" (say), do you mean 8 bits after the decimal point (i.e., 8 bits for the fractional part, plus an unlimited number of bits for the integer part); or 8 bits total? If the latter, the answer will depend on the quantization scheme, i.e., where you draw the bins. You could quantize to the nearest integer; to the nearest even integer; to the nearest half-integer; etc.; and each will lead to a different answer. $\endgroup$
    – D.W.
    Commented Sep 29, 2017 at 15:45
  • $\begingroup$ @D.W. Well there is no decimal point. An ADC outputs a unitless binary number ranging from 0 to 2^n-1 where n is the resolution in bits. Yes there's an equivalent range of possible readings represented by 0 and 2^n-1, but this is part of the question. Or the problem - I'm struggling. I can measure this entropy empirically in 15 minutes on my bench but forming a theoretical question is hard. It's all about the quantization levels as you suggest. Help! $\endgroup$
    – Paul Uszak
    Commented Sep 29, 2017 at 16:00
  • $\begingroup$ My point is that the quantization scheme determines how you map a real number to a number from 0 to 2^n-1. There are many ways to do that. Typically, that is done by mapping $x$ to $\lfloor x/c \rfloor$, for some constant $c$. However, you have to pick a constant $c$. A different value of $c$ will lead to a different mapping (a different quantization scheme), and thus will lead to a different answer to your question (a different value for the entropy of the quantized value), so you need to tell us the value of $c$. Also, you haven't answered my first question about "recorded entropy rate". $\endgroup$
    – D.W.
    Commented Sep 29, 2017 at 16:07
  • $\begingroup$ @D.W. The answer is almost here (csrc.nist.gov/csrc/media/events/…) but it's a presentation without the formulae. $\endgroup$
    – Paul Uszak
    Commented Sep 29, 2017 at 16:19


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.