# Questions tagged [entropy]

The tag has no usage guidance.

89 questions
Filter by
Sorted by
Tagged with
30 views

### Proving an entropy inequality

I am given that $Z$ is independent of $(X,U)$, where $Z$ and $X$ are binary random variables while $U$ is an arbitrary random variable. I need to prove the following: $$H(X\oplus Z|U) \geq H(X|U)$$ ...
22 views

136 views

### Collision entropy definition

The collision entropy is defined as the Renyi entropy for the case $\alpha = 2$. It is given by $$\mathrm{H}_{2}(X)=-\log \sum_{i=1}^{n} p_{i}^{2} \tag{1}$$ Take two random variables $X$ and $X'$ ...
8 views

### Categorical crossentropy on one hot encoded 6 dimensional target columns as output classes in keras

In keras , I have 6 targets which I one hot encoded , now I have 6 column of either 1 or 0. When I used categorical crossentropy I get 1% validation accuracy and when I used binary crossentropy I get ...
97 views

72 views

### Is it possible for a [futuristic] computer to contain the information of the whole universe?

(This is a question of information theory, data compression and entropy, so I believe it fits CS forum) Does the fact that the computer itself is a part of the universe make it logically impossible ...
68 views

### What is the relationship between entropy rate and quantization?

I have a totally random source of signal data that looks like a typical normal distribution. I've included an image as I like pictures:- The source has a mean of 0, and a standard deviation of 1. ...
142 views

### Compression of gaussian variables

Say I have 2 Gaussian sources X and Y. They are generated with mutivariate gaussian distribution with ...
188 views

167 views

### Can Von Neumann bit extraction be made more efficient?

I want to develop a previous question regarding Von Neumann debiasing /randomness extraction. The typical solution (as posted) is to take pairs of throws and output a bit based on a comparison of ...
141 views

### Kraft's inequality and Shannon's noiseless coding theorem for an encoding

A discrete memoryless source W has words $w_1,w_2,w_3,w_4,w_5,w_6$ that occur with probablilities $0.05,0.05,0.15,0.2,0.25,0.3$ respectivley. Does there exist a compact instantaneous binary encoding ...
67 views

### Approximate conditional entropy

Given a set of random variables $X = \{x_1, x_2, \dots, x_n\}$. If the conditional entropy for all $Y \subset X - \{X_i\}$ where $|Y| \leq 5$. How to approximate conditional entropy when $|Y| = 10$ ...
119 views

### Extracting Randomness from Mouse Acceleration

I'm working on trying to make an "entropy pool" that will be fed as input into an RNG (as in, ex, Fortuna). In order to do so, I need to take various collected data and extract as much entropy as ...
83 views

### Numerical example of theoretical diff file size using Kullback Leibler?

I understand that the theoretical size of a diff patch between two similar files can be calculated using Kullback Leibler (KL) as described @ Wikipedia. Can anyone point me to a numerical example of ...
I'd like to measure how much information a document $D$ contains. Clearly, the New York Times published yesterday contains more information than my diary wrote on the same day. But, I do not know ...