# Source entropy and other questions related to information theory

Kolmogorov-Sinai entropy (KS) explains the mathematical concept behind KS entropy.

$$h ( T ) =\sup\limits_{\xi} \, h ( T , \xi )$$

defines the formula for KS where the left-hand side is nothing but the Shannon's entropy. In many papers I have seen that KS is defined as the supremum over source entropy. These conflicting views raises the following questions

1. What is the difference between Shannon's entropy and source entropy (source entropy given by $\lim \frac{1}{n}\cdot h(T)$ where $n$ is the length of the word or sequence,unsure though).
2. Would source entropy be the entropy of the raw signal before it has been quantized or should it be that of the quantized?

3. Is 'n' the length of the data series or the number of quantization levels>

4. Topological entropy (TS) explains topological entropy whose formula looks exactly as KS. So, what is the difference since KS is also known as metric entropy and not known as Topological entropy?
• Crossposted on Signal Processing. Please don't do that.
– Raphael
Jul 25 '12 at 6:31
• From your first link: "The notion of Metric Entropy of dynamical system, also known as Measure-Theoretic Entropy, Kolmogorov Entropy, Kolmogorov-Sinai Entropy, or just KS entropy," Does that answer 3.?
– Raphael
Jul 25 '12 at 6:36
• Thank you Raphael,yes it does answer and do not know how I missed that!! Jul 25 '12 at 6:42
• In the light of this and the fact that 4. is offtopic here, please edit your question to focus on 1. and 2. (and ideally on only one of them).
– Raphael
Jul 26 '12 at 6:51
• Here, $n$ is the length of the data series. Oct 23 '12 at 4:33

If you look at wikipedia you see that topological entropy is modeled after Kolmogorov-Sinai (KS) entropy, but it is defined without reference to a probability measure $\mu$ on the space.
For KS entropy, you need to use Shannon entropy: $-\sum_i\mu(X_i) \log \mu(X_i)$ for some partition of the space $\{X_i\}$, which can only be defined if you have some probability measure $\mu$ on your space, while topological entropy is defined without reference to a probability measure.
Since putting different probability measures on the space will give different KS entropies, while not affecting the topological entropy, these entropies are different. However, they are related by the theorem that the topological entropy is the supremum of the Kolmogorov-Sinai entropy over all probability measures $\mu$.