I am reading PCA in Bishop's Pattern Recognition and Machine Learning pg. 562, here in Lagrange multiplier but I don't get it in the highlighted: I wonder is it $u_1$ or $u^T_1$?
Your help is appreciated!
$\begingroup$
$\endgroup$
-
$\begingroup$ The title you have chosen is not well suited to representing your question. Please take some time to improve it; we have collected some advice here. Thank you! $\endgroup$ – Raphael♦ Apr 29 '18 at 15:38
-
$\begingroup$ Don't use images as main content of your post. This makes your question impossible to search and inaccessible to the visually impaired; we don't like that. Please transcribe text and mathematics (note that you can use LaTeX) and don't forget to give proper attribution to your sources! $\endgroup$ – Raphael♦ Apr 29 '18 at 15:38
$\begingroup$
$\endgroup$
You should take the gradient of the function, that is, the vector of partial derivatives with respect to all coordinates of $\mathbf{u}_1$, and equate it to the zero vector. Let us suppose that the individual coordinates of $\mathbf{u}_1$ are $u_1^1,\ldots,u_1^n$, so that (12.4) reads $$ \sum_{i=1}^n \sum_{j=1}^n S_{ij} u_1^i u_1^j + \lambda_1 \left(1 - \sum_{i=1}^n (u_1^i)^2\right). $$ The derivative with respect to $u_1^t$ is $$ \sum_{i=1}^n (S_{it} + S_{ti}) u_1^i - 2\lambda_1 u_1^t = 2 (\mathbf{S} \mathbf{u}_1)_t - 2\lambda_1 u_1^t, $$ since $S$ is symmetric. The gradient is thus $$ \nabla (12.4) = 2\mathbf{S} \mathbf{u}_1 - 2\lambda_1\mathbf{u}_1. $$ Equating it to zero, we obtain the condition $\mathbf{S} \mathbf{u}_1 = \lambda_1 \mathbf{u}_1$, that is, $\mathbf{u}_1$ is an eigenvector of $\mathbf{S}$ corresponding to the eigenvalue $\lambda_1$.
answered Apr 29 '18 at 15:35
