I know Shannon Entropy is defined for messages. When it's said the size of a file stored in a HDD is 1 MB, are we talking about the Shannon Entropy? If so, how do we extend the definition to static information (not flowing like messages)? If not, how come they have the same unit, i.e. bits?
5
-
1$\begingroup$ Do mean what Windows tells you? No, that's way more than the entropy. Which would be useless: you don't want to know how much information got stored, but how much space is occupied. $\endgroup$– RaphaelCommented Apr 17, 2018 at 11:03
-
$\begingroup$ If we compress the file to a Zip file, or maybe compress it as much as possible, do we get the entropy? $\endgroup$– asmaniCommented Apr 17, 2018 at 11:13
-
1$\begingroup$ Still no. Every filesystem in practical use stores data with overhead. Read up on block size in this context. Also, zipping is not perfect; it doesn't even always increase entropy! Shannon entropy is a purely theoretical ideal. $\endgroup$– RaphaelCommented Apr 17, 2018 at 12:04
-
$\begingroup$ Now theoretically, is there an entropy associated with every digital file? Suppose we have an 10x10 image. If all the 100 pixels have the same RGB, it should have less entropy than if the pixels had random RGB. Can we calculate the entropy of such file? If so, how can we apply entropy to static files sitting on a HDD? $\endgroup$– asmaniCommented Apr 17, 2018 at 15:48
-
1$\begingroup$ The Kolmogorov complexity of the file content when interpreted as a string may be closer to the measure you're looking for. Shannon entropy appears out of place here. $\endgroup$– KaiCommented Apr 18, 2018 at 10:07
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
No. That is telling you the size of the file, i.e., that's a statement about its length, not its entropy.
Of course, the length gives an upper bound on the entropy, but it is not itself a value for the entropy. The actual entropy could be smaller.
No, compressing the file does not tell you its entropy. The length after compression is, on average, a reasonable upper bound for the entropy -- but the true entropy could be much smaller.
Beware that it only makes sense to talk about the entropy of a random process or the entropy of a distribution (e.g., the entropy of a process used to generate a file), not the entropy of a single file. As an analogy, consider this old joke:
It doesn't make sense to say that "the number nine is random"; that's a property of the process used to generate the number, not the number itself. You could get a 9 by rolling a ten-sided die (which is a random process), or you could get a 9 by writing the largest single-digit decimal number (that's not a random process). Similarly, it doesn't make sense to ask what is the entropy of the number nine; that is not well-defined. We can talk about the entropy of the random process that was used to generate it, though.
See a good textbook to learn more about the subject of entropy.
-
$\begingroup$ Thanks. I find your answer a little contradictory. You already talked about the entropy of a file, but then you say it's nonsense? $\endgroup$– asmaniCommented Apr 17, 2018 at 22:57
-
1$\begingroup$ @Asmani, I'm sorry to hear that you find it confusing. Perhaps you will understand after learning more about entropy, or perhaps someone else will be able to explain more to you. When I say "that's not a statement about its entropy", that's a good thing, because if it was an attempt to make a statement about its entropy, it would be meaningless/ill-defined. $\endgroup$– D.W. ♦Commented Apr 17, 2018 at 23:00
-
$\begingroup$ "the entropy of the random process that was used to generate it". Will this give a unique amount of entropy, or does it depend on how the process is done? $\endgroup$– asmaniCommented Apr 17, 2018 at 23:31
-
$\begingroup$ (I had a course on digital telecommunications which covered information theory to some levels. But well, it was about 5 years ago, and I remember we were only dealing with signals, not stored data.) $\endgroup$– asmaniCommented Apr 17, 2018 at 23:31