Questions tagged [information-theory]

Questions about Information theory, entropy, and information content of various sources

41 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
199 views

Average redundancy in Huffman or Hu-Tucker codes on random symbol probabilities

Huffman and Hu-Tucker codes are well-known compression schemes, which both come close to the entropy lower bound. It is known that if $L_1$ and $L_2$ are the lengths of a Huffman resp. Hu-Tucker code, ...
5
votes
0answers
62 views

Name of a type of code similar to block codes

I've encountered a system where I need to construct a sort of quasi block code: We want to encode a symbol $s$ from a finite-sized alphabet $\mathcal{S}$ using $N$ segments of information. The $i^{...
4
votes
0answers
40 views

Understanding simulated annealing information theoretically

So I recently rediscovered simulated annealing though a path that others seem to be well aware of. I was aware of Metropolis-Hastings as a sampling algorithm that creates a Markov-Chain who's ...
4
votes
0answers
117 views

channel capacity of a general binary channel

I have found worked examples for special cases of binary channel such as the binary symmetric channel and the Z-channel. However, I am interested in a more general type of binary channel $X \to Y$ in ...
3
votes
0answers
129 views

How to compare conditional entropy and mutual information?

I am solving a problem of information theory. The problem reads, Consider a stationary memoryless channel specified by the channel matrix $T = \begin{pmatrix}1-q&q\\r&1-r\end{pmatrix}$. ...
2
votes
0answers
44 views

What is the compressibility of this simple “book”?

Compressibility is defined as $$C=\frac{2^{HN}}{2^{H_{max}N}}$$ The book is made up of a simple alphabet of only {a,b,c,d} which occur with probabilities $$P(a)=0.2, P(b)=0.4, P(c)=0.1, P(d)=0.3$$ ...
2
votes
0answers
42 views

Computing/sketching essential bit content of a binary source

I'm given the Bernoulli distribution of a biased coin toss with probability distribution $P_X = \{0.2, 0.8\}$ over the alphabet $\mathcal{A}_X=\{0,1\}$. I want to sketch the normalized essential bit ...
2
votes
0answers
26 views

Classify/Distinguish between 8008 binary grids, with 13 queries

I have $8008$ binary grids of size $6 \times 10$ (they are all grids with the property described below), which I want to distinguish between with at most $13$ queries. A query will determine if the ...
2
votes
0answers
31 views

How to know a certain grammar is parse-able

Is it possible to parse all kinds of structured data and give them a semantic meaning? For example, C++ is a really complicated language and I could never imagine a parser would be possible for it. ...
2
votes
0answers
27 views

Is there a universal metric of “size of a program”?

There is a universal metric of information: amount of bits. It's universal in the sense that if we write a piece of information in DNA (4-ary digits), we can simply multiply by 2-log-4 to get the ...
2
votes
0answers
272 views

The meaning of ontology in computer science

I am reading a paper and ran into this word Ontology-based, What does it really mean? I did searching about it but still ambiguous to me, can someone elaborate this ...
2
votes
0answers
50 views

Clustering with probabilities / vector quantization with arbitrary distance measures

Suppose I'm given $n$ points $x_1,\dots,x_n$ in some space $\mathcal{S}$ (think: $\mathbb{R}^d$), and probabilities $p_1,\dots,p_n$ that form a probability distribution (so $p_1 + \dots + p_n=1$). ...
2
votes
0answers
88 views

Notions of computational hardness in terms of information flow?

If we consider polynomial-time (or log-space) computable reductions $<_p^m$ as transformations between computational problems, then the following definitions of known complexity classes suggest ...
1
vote
0answers
25 views

Can we think of information theory in terms of “a measure on set of information”?

In information theory, we deal with the quantities $I(X;Y), H(X),H(Y), H(X|Y), H(Y|X)$. These are just numbers, but I intuitively think of them as the "measure" of a set of information. There is at ...
1
vote
0answers
21 views

How connected are information theory and algorithmic information theory?

In the book by Cover and Thomas on information theory, there is a chapter on algorithmic information theory (kolmogorov complexity and so forth). As far as I understand, algorithmic information ...
1
vote
0answers
13 views

Example of channel where capacity is achieved without a uniform distribution on the output alphabet

The capacity of a discrete memoryless channel is given by the maximum of the mutual information over all possible input probability distributions. That is \begin{align} C &= \max_{p_X} I(X:Y) \\ &...
1
vote
0answers
15 views

Do all Cellular Automata have some kind of information boundary? Can all Cellular Automata be modelled with the Bekenstein Bound?

Since they are discrete models, do they have some kind of information boundary? Can all Cellular Automata models be related to the Bekenstein Bound? https://en.wikipedia.org/wiki/Bekenstein_bound
1
vote
0answers
37 views

How to use HITS algorithm to rank pages

I am trying to implement a simple web search engine mechanism with using HITS algorithm. I understand how the algorithm work and produce hubs and authority values for each page. But how can I use this ...
1
vote
0answers
49 views

How an insertion can be decoded in Varshamov and Tenengolts (VT) codes

I have a good idea of how deletion can be decoded in VT codes using Levenshtein algorithm. However, I don't have any idea how can an insertion be decoded? Can anybody give me a small example using a ...
1
vote
0answers
112 views

Second-order Markov text generation?

Looking at this video starting at 1:45, the author claims to be using a second-order approximation for a Markov text generation. He has one letter which he outputs followed by another letter which ...
1
vote
0answers
88 views

Find minimum conditional entropy

Task : Given $X$ random variables. Find out the minimum conditional entropy for a variable $x_i \in X$ when $x_i$ is conditioned upon any combination $k$ remaining variables. Find $min(Entropy (x_i | ...
1
vote
0answers
33 views

Information content of computational problems

The notion of low information content is used to describe sparse sets and tally sets in complexity theory. Such sets can not be $NP$-complete unless $P=NP$. I am not aware of a formal information-...
1
vote
0answers
24 views

Can independence numbers of box products of cycles increase after stabilizing?

Is there an evidence or a proof that the independence of strong products of graphs can increase after stabilizing? I am interested in odd cycles only. Let $C_n$ be an odd cycle and $\alpha(G)$ ...
0
votes
0answers
16 views

Broadcast channel that swaps output

Consider a broadcast channel that takes a pair of bits $(a_0, a_1)$ from Alice and transmits $a_0$ to Bob and $a_1$ to Charlie with probability $p$ and transmits $a_1$ to Bob and $a_0$ to Charlie with ...
0
votes
1answer
23 views

Parity vs Parity bit vs Parity sum

Assume that $B_b$ denotes the finite set of bitstrings of length $b$, if we are given its subset $A = \{e_i\}, i \in \{0,... n\}$, such that $e_i \in B_b$, what is "the parity sum of As bitstrings" ($...
0
votes
0answers
24 views

Information Theory: Comparing surprisal of words with varying count frequency

This is a very broad question, I'm not sure if cstheory is the better place. How can I compare the conditional surprisal of words that vary in frequency? $S(w|context)=−log(p(w|context))=−log(\frac{...
0
votes
1answer
48 views

information theory, find entropy given Markov chain

There is an information source on the information source alphabet $A = \{a, b, c\}$ represented by the state transition diagram below: a) The random variable representing the $i$-th output from this ...
0
votes
0answers
36 views

Is entropy a good indicator of the quality of a lossy compression?

Say I want to quantitively evaluate the effectiveness of several color-to-grayscale conversion algorithms, which can be considered as lossy compression. Would entropy be a good indicator? To ...
0
votes
0answers
21 views

go back n protocol scenario consequences

i am not sure with tihs question. i will post it here along with my attempt: the following scenario is regarding go back protocol n, with 3 bits for sequential numbering, assuming that the following ...
0
votes
0answers
56 views

finding the overhead and distance of an unknown code based on message making algorithm

for an information word M with m bits that is coded as following: M is coded into a word A using an unknown code that allows detection of not more than one error. the code word is the word obtained ...
0
votes
0answers
231 views

Huffman Coding vs Arithmetic Coding

Are there conditions under which given text $T$ that huffman code and arithmetic code will produce the exact encoding of $T$?
0
votes
0answers
10 views

Finding the 'capacity' of a network by treating it as a channel

Let's say I have a fairly basic network with two senders and two receivers. I'm wondering if I can treat this network as an effective communication channel that takes an input alphabet made up of the ...
0
votes
0answers
71 views

Is it possible for a [futuristic] computer to contain the information of the whole universe?

(This is a question of information theory, data compression and entropy, so I believe it fits CS forum) Does the fact that the computer itself is a part of the universe make it logically impossible ...
0
votes
0answers
27 views

Secrecy rate of eavesdropper channel?

I want to find secrecy rate of a main channel in case of Gaussian broadcast channel. e.g. I have a single transmitter and a receiver where eavesdropper wants to eavesdrop the communication over ...
0
votes
0answers
76 views

[Choosing tunable parameter M in Golomb/Rice Encoding]

I am a Computer Science student and I study Data Compression now. I read many methods but Golomb-Rice Encoding was not clear for me. I understand the method how it works but in the algorithm there is ...
0
votes
0answers
601 views

How does youtube do video quality change/video compression?

After uploading a high definition video in youtube, youtube automatically makes a low definition(standard definition) version of the video. I would like to know what algorithm they use to reduce the ...
0
votes
0answers
75 views

Is there a fundamental concept underlying trade-offs in CS and are they unavoidable?

There are many examples of trade-offs in computer science. The space-time trade-off is a well-known one. Often an increase in memory use can lead to faster execution time, and vice-versa. Caching ...
0
votes
0answers
74 views

If you want to archive (and compress) a number of files into one archive is it more efficient to archive first and then compress?

When compressing multiple files, is it more efficient to (1) first compress each individual file separately and then concatenate/merge the results, or (2) first concatenate/merge the files into a ...
0
votes
0answers
36 views

Fundamental representation of data relationships?

This is an offshoot of this question Which is more fundamental: key-value or subject-predicate-object? Is there any fundamental, hypothetical or practical, representation of data relationships? Both ...
0
votes
0answers
61 views

G1, G2, H1, H2 are submatrices of the generator and parity check matrices of a code as described below. Can G1 * H2' and G2 * H1' both be all zeroes?

$ G $ and $ H $ are the generator and parity check matrices respectively of a linear block code. Let $ G_1 = G(:, 1:n-s) $ (Matlab style representation of sub matrices). That is, $ G_1 $ is equal to ...
-1
votes
1answer
38 views

Integer Arithmetic Coding misunderstanding of parameters

Given this alphabet $\{a,b,c\}$ where $P(a) = 2/5$, $P(b) = 2/5$, $P(c) = 1/5$. Encode this string : $bcba$. I have to encode this using 5 bits. I have been looking in the Introduction to Data ...