I know Shannon Entropy is defined for messages. When it's said the size of a file stored in a HDD is 1 MB, are we talking about the Shannon Entropy? If so, how do we extend the definition to static information (not flowing like messages)? If not, how come they have the same unit, i.e. bits?

  • 1
    $\begingroup$ Do mean what Windows tells you? No, that's way more than the entropy. Which would be useless: you don't want to know how much information got stored, but how much space is occupied. $\endgroup$ – Raphael Apr 17 '18 at 11:03
  • $\begingroup$ If we compress the file to a Zip file, or maybe compress it as much as possible, do we get the entropy? $\endgroup$ – Asmani Apr 17 '18 at 11:13
  • 1
    $\begingroup$ Still no. Every filesystem in practical use stores data with overhead. Read up on block size in this context. Also, zipping is not perfect; it doesn't even always increase entropy! Shannon entropy is a purely theoretical ideal. $\endgroup$ – Raphael Apr 17 '18 at 12:04
  • $\begingroup$ Now theoretically, is there an entropy associated with every digital file? Suppose we have an 10x10 image. If all the 100 pixels have the same RGB, it should have less entropy than if the pixels had random RGB. Can we calculate the entropy of such file? If so, how can we apply entropy to static files sitting on a HDD? $\endgroup$ – Asmani Apr 17 '18 at 15:48
  • 1
    $\begingroup$ The Kolmogorov complexity of the file content when interpreted as a string may be closer to the measure you're looking for. Shannon entropy appears out of place here. $\endgroup$ – Kai Apr 18 '18 at 10:07

No. That is telling you the size of the file, i.e., that's a statement about its length, not its entropy.

Of course, the length gives an upper bound on the entropy, but it is not itself a value for the entropy. The actual entropy could be smaller.

No, compressing the file does not tell you its entropy. The length after compression is, on average, a reasonable upper bound for the entropy -- but the true entropy could be much smaller.

Beware that it only makes sense to talk about the entropy of a random process or the entropy of a distribution (e.g., the entropy of a process used to generate a file), not the entropy of a single file. As an analogy, consider this old joke:

It doesn't make sense to say that "the number nine is random"; that's a property of the process used to generate the number, not the number itself. You could get a 9 by rolling a ten-sided die (which is a random process), or you could get a 9 by writing the largest single-digit decimal number (that's not a random process). Similarly, it doesn't make sense to ask what is the entropy of the number nine; that is not well-defined. We can talk about the entropy of the random process that was used to generate it, though.

See a good textbook to learn more about the subject of entropy.

  • $\begingroup$ Thanks. I find your answer a little contradictory. You already talked about the entropy of a file, but then you say it's nonsense? $\endgroup$ – Asmani Apr 17 '18 at 22:57
  • 1
    $\begingroup$ @Asmani, I'm sorry to hear that you find it confusing. Perhaps you will understand after learning more about entropy, or perhaps someone else will be able to explain more to you. When I say "that's not a statement about its entropy", that's a good thing, because if it was an attempt to make a statement about its entropy, it would be meaningless/ill-defined. $\endgroup$ – D.W. Apr 17 '18 at 23:00
  • $\begingroup$ "the entropy of the random process that was used to generate it". Will this give a unique amount of entropy, or does it depend on how the process is done? $\endgroup$ – Asmani Apr 17 '18 at 23:31
  • $\begingroup$ (I had a course on digital telecommunications which covered information theory to some levels. But well, it was about 5 years ago, and I remember we were only dealing with signals, not stored data.) $\endgroup$ – Asmani Apr 17 '18 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.