What is the difference between distributed representations and factors of variation in deep learning

Question

In the field of deep learning, people often talk about factors of variation which, in my understanding (in terms of dimensionality reduction), are the latent variable directions capturing variability in the data. I also often come across the term distributed representations which are basically the features of the input data that are not mutually exclusive.

Sometimes for object recognition, one may wish to extract distributed features of the object that are invariant to small variations (which are a result of combination of many factors of variation).

On the other hand, in case of autoencoder, one needs to extract the factors of variation as completely as possible so as to minimize the reconstruction error.

In the above two cases, factors of variation seem to have two contradicting roles, nevertheless they are both referred to as useful features for intermediate layers of a deep architecture.

So what is the difference between factors of variation and distributed features? I am very confused.

Answer 1

Usually, factors of variation are independent from each other. However, sometimes in feature extraction, some combination of these factors can be helpful to detect better and accurate features to explain the behavior of the observation.

Therefore, factors of variation are used to explain distributed features, and their relation likes the relation between basis of a space and a vector which is defined in the space using these basis or prime factors.

Although, observations change base on the factors of variation and distributed features, distributed features change base on the factors of variation. In the other words, features of variation are prime factors, and distributed features are defined base on the prime factors.