The manifold hypothesis is the statement that real-world high dimensional data (such as images) lie on low-dimensional manifolds embedded in the high-dimensional space. It has been tested to be true quite extensively. So could someone explain or give some intuition why it is true? Why doesn't real world data such as dogs form manifolds of same dimension as the image space?
Presumably you don't really get a manifold.
For one thing, there's probably a nonzero chance of any particular coordinate being perturbed arbitrarily -- an impurity in a crystal, a grey hair on a head of brown ones, etc. -- so you'd really get something more like a probability distribution concentrated near a lower dimensional platonic ideal.
Second, there are probably all sorts of behaviors going on that aren't allowed in a manifold, like self-intersections, edges, and components of different dimensions.
That said, simple statements like "the left side of someone's face usually looks about like the mirror image of the right" cut the dimensionality of the space nearly in half. Add enough such independent conditions and you whittle things down quite a bit.
You know what they say about big hands? Big feet!
Correlations abound in real high-dimensional data. For instance, if our data set were 206 "length" measurements of human bones, we would definitely discover a correlation between the features.
In terms of images, similar colors tend to be around similar pixels, with "zero-dimensional" borders being the exception (zero-dimensional wrt to image, in that a border is a line in a plane). Consider that a random image would be static. In other words, the implicit organization of data that we are interested in forms a manifold in its space.
The manifold hypothesis is perhaps best seen as a heuristic. The practical observation is that dimension reduction often works well, particularly with sufficiently dense samples.
Intuitively, extrinsic dimensionality of a dataset often exceeds intrinsic dimensionality of an underlying phenomena. We tend to collect more data than is needed to identify a pattern. For example, one can often recognize a friend from far away, just based on a single feature, such as their posture or clothing style alone.
You mentioned it has been tested to be true extensively. A mathematical proof under certain strict conditions was given in "Testing the Manifold Hypothesis", a 2013 paper by MIT researchers, where the statistical question is asked
What is the number of samples needed for testing the hypothesis that data lie near a low dimensional manifold?. The desired result is that the sample complexity of the task depends only on the "intrinsic" dimension, volume, and reach, but not the “ambient” dimension. We approach this by considering the empirical risk minimization problem.
[...] The answer to this question is given by Theorem 1 in the paper. The proof of the theorem proceeds by approximating manifolds using point clouds and then using uniform bounds for k-means (Lemma 6 of the paper).
The results of this proof of course do not imply that every pattern can be seen in any lower dimensional representation. Rather, the hypothesis is merely intuitively apparent when the number of degrees of freedom exceeds the empirically determined intrinsic dimensionality.