I am reading PCA in Bishop's Pattern Recognition and Machine Learning pg. 562, here in Lagrange multiplier but I don't get it in the highlighted: I wonder is it $u_1$ or $u^T_1$?
Your help is appreciated!
Computer Science Stack Exchange is a question and answer site for students, researchers and practitioners of computer science. It only takes a minute to sign up.
Sign up to join this communityI am reading PCA in Bishop's Pattern Recognition and Machine Learning pg. 562, here in Lagrange multiplier but I don't get it in the highlighted: I wonder is it $u_1$ or $u^T_1$?
Your help is appreciated!
You should take the gradient of the function, that is, the vector of partial derivatives with respect to all coordinates of $\mathbf{u}_1$, and equate it to the zero vector. Let us suppose that the individual coordinates of $\mathbf{u}_1$ are $u_1^1,\ldots,u_1^n$, so that (12.4) reads $$ \sum_{i=1}^n \sum_{j=1}^n S_{ij} u_1^i u_1^j + \lambda_1 \left(1 - \sum_{i=1}^n (u_1^i)^2\right). $$ The derivative with respect to $u_1^t$ is $$ \sum_{i=1}^n (S_{it} + S_{ti}) u_1^i - 2\lambda_1 u_1^t = 2 (\mathbf{S} \mathbf{u}_1)_t - 2\lambda_1 u_1^t, $$ since $S$ is symmetric. The gradient is thus $$ \nabla (12.4) = 2\mathbf{S} \mathbf{u}_1 - 2\lambda_1\mathbf{u}_1. $$ Equating it to zero, we obtain the condition $\mathbf{S} \mathbf{u}_1 = \lambda_1 \mathbf{u}_1$, that is, $\mathbf{u}_1$ is an eigenvector of $\mathbf{S}$ corresponding to the eigenvalue $\lambda_1$.