# Questions tagged [information-theory]

Questions about Information theory, entropy, and information content of various sources

286 questions
Filter by
Sorted by
Tagged with
11 views

In my lecture, the lecturer introduced without much explanation that the following applies: Given a discrete random variable $X$ $N$ possible values: $x_{1}, x_{2}, \ldots, x_{N}$ Associated ...
7 views

### Is there an entropy evaluation method with an unified length?

For entropy, the most common one is Shannon entropy, however, it ignores time series of data. For instance, data 0x00001111 and 0x01010101 are given the same entropy. It is obvious that the second ...
11 views

### What is the difference between erasure coding and RAID if your erasure code is parity check?

I'm reading about erasure coding and saw that one of the erasure codes is parity check. As far as I can tell this is just a generalization of something like RAID5. If you select something like parity ...
20 views

### Proof in the "Reaching Agreement in the Presence of Faults"

I am reading the "Reaching Agreement in the Presence of Faults", M. Pease et al and trying to understand their proof for the $n \geq 3m+1$ case. In the induction step $m \gt 0$ it says the ...
11 views

### Information theory - Expurgation step to go from average error to worst case error in the large error regime

Consider a discrete memoryless channel $N$. We use a code to send messages over this channel. Shannon showed that if we have a code $C$ with a finite number of codewords $|C|$ such that the average ...
53 views

### What is the entropy of an unordered list?

I'm trying to compress unordered lists of a few thousand integers for transmission over HTTP, and Claude Shannon is disappointing me with his mathematical ambiguity :) Each integer is 6-digits, so ...
57 views

### Is every binary sequence the output of some meaningful-text-input algorithm?

Here is the problem: Let's say we have a random binary sequence, just an arbitrary sequence of zeroes and ones (of some arbitrary length of digits). Can we find an algorithm that would decode/...
35 views

### Is an AI upscaler incapable of reducing entropy?

I was reading the description of Anime4K (a video upscaler software) and I found a statement triggering my attention: [upscaling is done] without any meaningful decrease in entropy (lost information ...
61 views

### Deriving a lower bound on the conditional entropy, conditioned on an event

I came across Lemma 19 in Certifying Equality With Limited Interaction, which states the following for jointly distributed random variables $Z$, $W$, where $Z$ takes values in $\{0,1\}^n$, and some ...
13 views

### Asymptotically Optimal Universal Code In Other Bases

Universal codes are fairly well studied, and many asymptotically optimal universal codes exist for binary data (see https://en.wikipedia.org/wiki/Universal_code_(data_compression) especially https://...
48 views

### Capacity of Binary Erasure Channel

Consider the the binary erasure channel, with input and output alphabet $\{0,?,1\}$ and channel matrix \begin{bmatrix} 1-\lambda-\mu & \mu & \lambda\\ 0 & 1 & 0\\ \lambda & \mu &...
27 views

### Why does attempting to estimate the entropy of text by randomly choosing chars in it and counting how often they are equal give wildly wrong results?

Why does attempting to estimate the entropy of a string, by randomly choosing pairs of (not necessarily adjacent) characters in it, and counting how often the selected characters in the pairs are ...
29 views

### Essential bit content - prove that we can't use less bits than that

Let $H_0$ be $\log_2(|A|)$, where $A$ is a set. Let $C$ be a compressor $C\colon A \to \{1,0\}^l \cup \bot$. This is a silly question, because intuitively it seems obvious. How can I prove that $l$ ...
27 views

### Algorithmic information theory with stochastic algorithms?

Suppose we define a class of algorithms that is allowed to sample i.i.d. Bernoulli bitstrings of arbitrary length, and use these to generate outputs. If we are allowed to use algorithms like this, ...
19 views

### How Data Compression relates to Estimating Distribution?

I recently read this paper Mahoney, 1999. And encountered this line, optimal compression of a probabilistic language L with unknown distribution (such as English) using an estimated distribution M (...
41 views

### How to calculate the entropy of a system with multiple states

I'm stuck in trying to compute an overall entropy calculation with an agent. Let me first introduce some background of the problem. Basically, I'm doing some work with the contextual bandit problems. ...
81 views

### Information-theoretic lower bound for succinct string dictionary of the Unicode Name property

Background The literature on succinct data structures refers often to the “information-theoretic lower bound” of encoding data, i.e., the minimum number of bits needed to store the data – a concept ...
30 views

### Maximal prefix codes and maximal length

Let $X$ a maximal prefix code on an alphabet $A$, $m(X)$ its maximal length, $F = X \cap A^{m(X)}$ and $F’ \subseteq A^{m(X)}$. Let $X’ = X \setminus F \cup F’$ a maximal prefix code. Why is it true ...
123 views

### what is the relationship between entropy and variance?

Consider a simple Bernoulli variable X X = 1 with probability p X = 0 with probability (1-p) The variance is simply p(1-p). The ...
178 views

### Must a Turing machine tape be binary?

I once asked why does computer data bits are usually organized on binary (base 2) sets, rather than on unary (base 1) sets, aiming to also understand why its not also ternary (base 3), heptary (base 7)...
34 views

### Shared randomness does not increase capacity of a noisy channel - Why?

Why is it the case that when Alice and Bob use a noisy channel for communication, the capacity of the channel does not increase even if they are allowed to share pre-distributed randomness? This is ...
16 views

### Compressing the output of a discrete memoryless channel

Let $x\in \mathcal{X}$ be a symbol from an input alphabet, let $p(y|x)$ be a conditional probability distribution corresponding to a discrete memoryless channel and let $y\in\mathcal{Y}$ be an output ...
10 views

### Achieving the capacity of an AVC under random coding

Is there any work that shows how to achieve the random coding capacity in an Arbitrarily Varying Channel (AVC)? The best I can find is in the book 'Information Theory: Coding Theorems for Discrete ...
14 views

### How to interpret parametric formulation of information bottleneck?

I'm reading this paper on latent representations with the information bottleneck https://arxiv.org/pdf/1804.06216.pdf and in section three, the authors write that the parametric formulation of the ...
25 views

### How to build 4 codewords with a code distance of 5?

I wonder how can I construct 4 (distinct) codewords given the fact that code distance is 5. As far as I know that the code distance is the number of distinct bits between any 2 codewords. How to ...
52 views

### How to recognise the number of errors that can be detected and corrected of a large set of codewords (k) each having a specified number of bits (n)?

I am struggling to find how can I know the number of errors that can be corrected and detected using (n=10) bit code with a (k=550) codewords. As far as I know, to calculate the number of errors to be ...
13 views

31 views

### Intuitive explanation on why stochastic encoding performs better in channel coding

I am a little confused about stochastic encoding in channel coding. For example, in the identification problem (R. Ahlswede and G. Dueck, “Identification via channels”), the authors claim that we can ...
32 views

### Proving an entropy inequality

I am given that $Z$ is independent of $(X,U)$, where $Z$ and $X$ are binary random variables while $U$ is an arbitrary random variable. I need to prove the following: $$H(X\oplus Z|U) \geq H(X|U)$$ ...
105 views

24 views

### Complexity of maximization of entropy of Hamming distance of bitstrings

We have a set of possible "key"s $S$ represented by bitstrings of length $k$. In other words, $S$ contains an arbitrary subset of all bitstrings of length $k$. For example, when $k=3$, it can be \$S = \...
40 views

### on-the-fly decompress a flat-file database

I'm facing the following problem. I have a flat-file database (e.g. CSV). Since it's relatively large to store in memory, I'd like to compress it. Given a key, I need to return the uncompressed text (...
56 views

### Equivalence of two definitions of mutual information

I am learning quantum computing and as a background study, I am currently learning fundamentals of classical information theory. I thought it best to ask my doubts here. In Nielsen and Chuang, it is ...