Skip to main content
added 634 characters in body
Source Link
D.W.
  • 165.6k
  • 21
  • 230
  • 490

No. That is telling you the size of the file, i.e., that's a statement about its length, not its entropy.

Of course, the length gives an upper bound on the entropy, but it is not itself a value for the entropy. The actual entropy could be smaller.

No, compressing the file does not tell you its entropy. The length after compression is, on average, a reasonable upper bound for the entropy -- but the true entropy could be much smaller.

Beware that it only makes sense to talk about the entropy of a random process or the entropy of a distribution (e.g., the entropy of a process used to generate a file), not the entropy of a single file. See As an analogy, consider this old joke:

It doesn't make sense to say that "the number nine is random"; that's a property of the process used to generate the number, not the number itself. You could get a 9 by rolling a ten-sided die (which is a random process), or you could get a 9 by writing the largest single-digit decimal number (that's not a random process). Similarly, it doesn't make sense to ask what is the entropy of the number nine; that is not well-defined. We can talk about the entropy of the random process that was used to generate it, though.

See a good textbook to learn more about the subject of entropy.

No. That is telling you the size of the file, i.e., that's a statement about its length, not its entropy.

Of course, the length gives an upper bound on the entropy, but it is not itself a value for the entropy. The actual entropy could be smaller.

No, compressing the file does not tell you its entropy. The length after compression is, on average, a reasonable upper bound for the entropy -- but the true entropy could be much smaller.

Beware that it only makes sense to talk about the entropy of a random process or the entropy of a distribution (e.g., the entropy of a process used to generate a file), not the entropy of a single file. See a good textbook to learn more about the subject of entropy.

No. That is telling you the size of the file, i.e., that's a statement about its length, not its entropy.

Of course, the length gives an upper bound on the entropy, but it is not itself a value for the entropy. The actual entropy could be smaller.

No, compressing the file does not tell you its entropy. The length after compression is, on average, a reasonable upper bound for the entropy -- but the true entropy could be much smaller.

Beware that it only makes sense to talk about the entropy of a random process or the entropy of a distribution (e.g., the entropy of a process used to generate a file), not the entropy of a single file. As an analogy, consider this old joke:

It doesn't make sense to say that "the number nine is random"; that's a property of the process used to generate the number, not the number itself. You could get a 9 by rolling a ten-sided die (which is a random process), or you could get a 9 by writing the largest single-digit decimal number (that's not a random process). Similarly, it doesn't make sense to ask what is the entropy of the number nine; that is not well-defined. We can talk about the entropy of the random process that was used to generate it, though.

See a good textbook to learn more about the subject of entropy.

Source Link
D.W.
  • 165.6k
  • 21
  • 230
  • 490

No. That is telling you the size of the file, i.e., that's a statement about its length, not its entropy.

Of course, the length gives an upper bound on the entropy, but it is not itself a value for the entropy. The actual entropy could be smaller.

No, compressing the file does not tell you its entropy. The length after compression is, on average, a reasonable upper bound for the entropy -- but the true entropy could be much smaller.

Beware that it only makes sense to talk about the entropy of a random process or the entropy of a distribution (e.g., the entropy of a process used to generate a file), not the entropy of a single file. See a good textbook to learn more about the subject of entropy.