Comparing the entropy of a painting to a photograph

Question

This is a quote from the book, "The Three-Body Problem" (page 112 in the paperback):

He remembered taking a class in information theory as a third-year student in college. The professor had put up two pictures: One was the famous Song Dynasty painting Along the River During the Qinming Festival , full of fine, rich details; the other was a photograph of the sky on a sunny day, the deep blue expanse broken only by a wisp of cloud that one couldn't even be sure was there. The professor asked the class which picture contained more information. The answer was that the photograph's information content - its entropy - exceeded the painting's by one or two orders of magnitude.

The painting (link in quote, source is Famous Chinese Paintings) is in fact, very large - 10,671x500, or 5,335,500 pixels, as 21MB uncompressed PNG. No idea what size the photograph is...

So my question is, is the professor's statement correct - is the information-content likely two orders of magnitude larger in the photograph?

Programmer here, not a physicist computer-scientist, but my (limited) understand of entropy, is that it's a measure, not just of information, but unpredictable information. That is, if the picture were all exactly one shade of blue, it's entropy, or "information content", would be very low. And one way of measuring information content is compressibility (as a file, not a gas :) .

Given a file of slightly-varying blues (and some white clouds), my intuition is that it would be very compressible (some algorithms more than others, granted).

Would that reduce the "entropy" in the photograph, to something closer than the painting? Or would it still be two orders of magnitude higher?

Answer 1

The images in Does a simple photograph contain more information than a complex painting? - Signal Processing Stack Exchange aren't the originals, they have been processed by imgur, are .png files, and are only 762×633 pixels in size.

Answering based on that information is difficult, but even so, if one takes a section of the blue sky and zooms in on it one can see many patterns of many different colours, not only blue tones. The colour of any specific pixel isn't very predictive of that of its neighbours.

The painting has only a few different colours, and many regular patterns. It has many pixels with identically coloured neighbours. (Each brushstroke would cover a large number of pixels with the exact same colour of paint.)

So yes, without knowledge of what is significant, it's not unreasonable to conclude that the blue sky contains far more information. And this would have been even more so for the original high resolution photograph.

Answer 2

It sounds debatable to me. I could imagine a sense in which the quote might be correct, and another sense in which the quote might be considered incorrect.

One perspective would be: how many bits does it take to compress the image, using (perfect, ideal) lossless compression. Then I'm not sure how that would turn out, but it's possible that the compressed photograph would be larger (e.g., due to noise in the LSBs of the pixel intensities and the larger number of pixels). Modern cameras may have up to 60M pixels, compared to the 21M pixels in the painting, so it's possible that the compressed photograph might be larger than the compressed painting. So, in that sense, it's possible that the photograph might be said to have more information. (Of course, much of that information might be noise in the least significant bits of the pixels of the photograph, which isn't information that humans are likely to care about very much, but from a pedantic perspective, still counts as information.)

Another perspective would be: how many bits does it take to compress the image, using lossy compression, to some quality level. Then I would expect that the compressed image of the painting would absolutely be larger (after compression) than the compressed photograph, so in that sense, the painting can be said to have more information. (The discrepancy between lossless compression vs lossy compression is because lossy compression is likely to remove a lot of information that isn't perceptually important to humans.)

The distinction between the two perspectives amounts to whether we consider the pixel intensity as a discrete value (in which case entropy is usually considered to represent how many bits it would take to communicate those values exactly) or as a continuous value (in which case entropy is usually considered to represent how many bits it would take to communicate those values up to some acceptable amount of distortion/error). At a technical level, you can compare the definition of entropy for a discrete random variable, $H(X) = -\sum_x p(x) \log p(x)$, with the definition of entropy for a continuous random variable, $H(X) = - \int_x f(x) \log f(x) \; \text{d} x$ (more precisely we should compare to the rate from rate-distortion theory, but this can be approximated by the continuous entropy minus a constant that depends on the amount of distortion that can be tolerated). Both are called "entropy", even though they are slightly different, and both are considered a measure of the amount of information present, even though they are measured slightly differently.

Overall, if I were sitting in the audience, I wouldn't be too impressed just from that description alone, and I might suspect that the professor is trying to make a point that might be true but only in a limited, technical way; or I would suspect the professor is trying to be deliberately provocative (even if possibly slightly misleading) to get everyone's intention, and I'd reserve judgement until I saw where the professor is about to go with this. But that's just me.

From a pedantic perspective, the professor was wrong to talk about the entropy of a single image, as that is not well-defined. From a mathematical perspective, information is only defined as the amount of entropy in a random variable. So, you can meaningfully talk about the entropy in a random process, not in a single sample from that process. Mathematics provides a well-defined notion of entropy, which we often use as our way of measuring information. "Information", as a word, is ambiguous. If the professor was using the word "information" in the mathematically precise sense of entropy, then the professor was incorrect to speak about the information of a single image. Alternatively, sometimes we use the word "information" in a fuzzy sense that has no precise mathematical definition. Then it's not necessarily wrong to talk about the information in a single image, it is just not clear exactly what the speaker means by that, and it is not clear whether it is meaningful to talk about the information in a single piece of data (e.g., a single image). In that case, I think a student who wanted to be pedantic would be within their rights to ask the professor to be precise about exactly what they mean by "information content". However, I read the quote as implying that by "information", the professor actually means "entropy", and in that case, the professor's statements were problematic. (See also https://i.sstatic.net/ooy9J.gif and How best to statistically verify random numbers?)