From what I understand data entropy controls the limit of data compression and it depends on the probability of the characters in the file. Assuming that we have a file of size 256 bytes containing 256 different symbols and we made a bigger file consisting of many copies of that file, And because the small file has the maximum entropy a file of its size can have, the big file will also have that high entropy (because all characters have the same probability). Although that file has high entropy it still can be compressed to a very smaller size. What did I misunderstand here?


1 Answer 1


Entropy is a property of the source of the data, i.e., of the distribution that the data is drawn from -- not a property of the particular file. You can't calculate the entropy given a single file. Instead, you need to know the distribution that file came from. Or, to put it another way, the entropy is a measure of the process used to generate the particular file, not a measure of one particular file you happened to get from that process; and observing one output from that process isn't enough to let you calculate the entropy of the process. So, the question isn't well-formed.

You might be thinking of forming the empirical distribution of characters, then assuming the file came from a memoryless iid source (where each character was drawn independently according to that distribution), and then calculating the entropy under that assumption. But that assumption doesn't hold for your bigger file, so this isn't a valid way of calculating the actual entropy of the process that was used to generate the bigger file.

To learn more: How to practically measure entropy of a file?, https://en.wikipedia.org/wiki/Entropy_(information_theory).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.