Let f be a one-layer neural network which is linear (ie. no activation function). Let it have $p$ inputs and $q$ outputs. These are fully connected by weights $W$. We have $n$ inputs $x \in \mathbb{R}^d$ and $n$ outputs $y \in \mathbb{R}^{d'}$. We arrange these (column) vectors into matrices $Y$ and $X$, and the weights into a matrix, so that we can express the computation of the network on the whole dataset as $WX = Y$ (that is, if we could find the perfect $W$). Just to clarify, the network contains no hidden layer, just the input and output layers.

Training the network is a matter of choosing $W$ to minimize $e_W = \sum_{i=1}^n||y_i - Wx_i||^2$.

It seems to me that this problem should have a closed form solution that can be solved by a linear solver in some way. Unfortunately, my linear algebra is lacking. Is there a closed form solution? If not, where is the difficulty?


Yes, there is a closed form solution.

In the most general terms, $WX = Y$ is a linear equation, so it can be solved as $W = X^{-1}Y$. If $X$ has no inverse, using the pseudo-inverse $X^\dagger = X^T(XX^T)^{-1}$ will give the $W$ that minimizes $\sum_i||Wx_i - y_i||^2$.

For practical computation, it's best to compute the pseudo-inverse from a matrix decomposition, rather than multiplying the matrices explicity. The safest option is the singular value decomposition. The wikipedia article details the procedure.

The principle behind this solution used only basic calculus. Calling the error $E$, we set $\frac{\partial E}{\partial w_{ij}} = 0$ and solve for $w_{ij}$. This is detailed in [1, pages 89–93) and [2], both available online.

Alternative phrases for this problem are multivariate linear regression and affine registration.

[1] Neural Networks for Pattern Recognition, Christopher M. Bishop.

[2] Fitting affine and orthogonal transformations between two sets of points, Helmuth Späth, 2004


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.