I am trying to understand how PCA works, and think I got most of it except...

By calculating eigenvalues/vectors of the covariance matrix of the original dataset allows to find those dimensions where data is least correlated (lowest covariance). Sorting them based on eigenvalues is the method which allows the sorting of dimensions.

Hence, multiplying the original dataset by the 'feature matrix' allows to discard those dimensions which are least significant.

What I don't understand is the relation between the covariance matrix and eigenvectors. Why is it that eigenvectors of the covariance matrix automatically allow us to find the directions where data is least correlated?

Hope this makes sense :)


1 Answer 1


Each eigenvector of the covariance corresponds to a new feature (which is a combination of the original features).

The principal eigenvector of the covariance matrix corresponds to the direction of maximum variance (not least as mentioned in the question). Subsequent eigenvectors can be computed one by one under the constraint that they are orthogonal to the ones already computed. The fact that they are orthogonal makes the new features uncorrelated with each other. In other words, a set of $k$ eigenvectors of the covariance corresponds to a set of $k$ new features that are uncorrelated with each other.

There are many different viewpoints on what exactly principal components are. In general, these seemingly different viewpoints have strong connections with each other. For example, the first $k$ eigenvectors (i.e., the eigenvectors corresponding to the $k$ largest eigenvalues of the covariance matrix) capture the maximum cumulative variance of the data. At the same time they form a basis for a $k$-dimensional subspace (the space of new features) such that if you project your original data (each datapoint individually) on that subspace (i.e., if you optimally approximate each datapoint using the new features), then the total euclidean distance between the original datapoints and their projections is the minimum possible; by minimum we mean in comparison to any other $k$-dimensional subspace.

This short article takes a look at what PCA is from a few different viewpoints.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.