# MAP estimation (for stationary iid gaussian environment)

This is my first post, and have been self studying Haykin's Neural Networks and Learning Machines book. I'm not sure if this is a typo or if I'm doing something wrong, but I've been stuck on a statement on page 75.

The context was the process of finding a Maximum likelihood estimate for a parameter vector assuming a stationary, iid, and Gaussian environment.

Given: d is a scalar, i indexes time from 1 to N, and $\mathbf{w},\mathbf{x}_i$ are m-dimensional vectors.

$\hat{w}_{MAP} = max_{\mathbf{w}}[-\frac{1}{2}\Sigma_{i=1}^{N}(d_i - \mathbf{w}^T\mathbf{x}_i)^2 - \frac{\lambda}{2}||\mathbf{w}||^2]$

we should get the same result if instead we find the minimum of the quadratic function

$E(\mathbf{x}) = \frac{1}{2}\Sigma_{i=1}^{N}(d_i - \mathbf{w}^T\mathbf{x}_i)^2 + \frac{\lambda}{2}||\mathbf{w}||^2$

Haykin says you should arrive at the result (by differentiation wrt $\mathbf{w}$ and setting the result to 0)

$\hat{w}_{MAP} = (R_{xx} + \lambda I)^{-1} r_{dx}$

where $R_{xx} = -\Sigma_{i=1}^N \Sigma_{j=1}^N \mathbf{x}_i \mathbf{x}_j^T$ is the M*M correlation matrix of x and $r_{dx} = -\Sigma_{j=1}^N \mathbf{x}_i d_i$

My question is why isn't $R_{xx} = -\Sigma_{i=1}^N \mathbf{x}_i \mathbf{x}_i^T?$ Haykin goes through about a paragraph saying how the correlation matrix is time averaged and I think this is an important point. Thanks for any help!

• You are right; to minimize $E(x)$ as described above, then $R_{xx}$ is the matrix you describe (without the minus). Apr 9, 2015 at 0:48

$\newcommand{\bd}{\mathbf{d}}\newcommand{\bw}{\mathbf{w}}\newcommand{\bX}{\mathbf{X}}$Your observation is correct--it appears to be several typos. To see why let's re-write the likelihood equation to something a bit more suitable for differentiating using matrix/vector notation. $$\text{NLL}(w) = \frac{1}{2}||\bd - \bX\bw||^2_2 + \frac{\lambda}{2}||\bw||^2_2,$$
where $\bX$ is our matrix of observations, $\bd$ is our vector of labels and $\bw$ is our vector parameter of interest. Taking the derivative yields, $$\frac{\partial \text{NLL}}{\partial \bw} = -\bX^t(\bd - \bX\bw) + \lambda\bw.$$
Setting this equal to 0 and solving gives, \begin{align*}\lambda \bw &= \bX^t(\bd - \bX\bw) \\ &= \bX^t\bd - \bX^t\bX\bw \iff\\ \lambda\bw + \bX^t\bX\bw &= \bX^t\bd \iff \\ (\lambda\mathbf{I} + \bX^t\bX)\bw &= \bX^t\bd \iff \\ \hat{\bw}_{\text{MAP}} &= (\lambda\mathbf{I} + \bX^t\bX)^{-1}(\bX^t\bd). \end{align*}
Notice that $$\bX^t\bX = \sum_i \mathbf{x_i} \mathbf{x_i}^t = R_{xx}$$ and $$\bX^t\bd = \sum_i \mathbf{x_i} d_i = r_{dx}.$$