4
$\begingroup$

Let f be a one-layer neural network which is linear (ie. no activation function). Let it have $p$ inputs and $q$ outputs. These are fully connected by weights $W$. We have $n$ inputs $x \in \mathbb{R}^d$ and $n$ outputs $y \in \mathbb{R}^{d'}$. We arrange these (column) vectors into matrices $Y$ and $X$, and the weights into a matrix, so that we can express the computation of the network on the whole dataset as $WX = Y$ (that is, if we could find the perfect $W$). Just to clarify, the network contains no hidden layer, just the input and output layers.

Training the network is a matter of choosing $W$ to minimize $e_W = \sum_{i=1}^n||y_i - Wx_i||^2$.

It seems to me that this problem should have a closed form solution that can be solved by a linear solver in some way. Unfortunately, my linear algebra is lacking. Is there a closed form solution? If not, where is the difficulty?

$\endgroup$

1 Answer 1

5
$\begingroup$

Yes, there is a closed form solution.

In the most general terms, $WX = Y$ is a linear equation, so it can be solved as $W = X^{-1}Y$. If $X$ has no inverse, using the pseudo-inverse $X^\dagger = X^T(XX^T)^{-1}$ will give the $W$ that minimizes $\sum_i||Wx_i - y_i||^2$.

For practical computation, it's best to compute the pseudo-inverse from a matrix decomposition, rather than multiplying the matrices explicity. The safest option is the singular value decomposition. The wikipedia article details the procedure.

The principle behind this solution used only basic calculus. Calling the error $E$, we set $\frac{\partial E}{\partial w_{ij}} = 0$ and solve for $w_{ij}$. This is detailed in [1, pages 89–93) and [2], both available online.

Alternative phrases for this problem are multivariate linear regression and affine registration.

[1] Neural Networks for Pattern Recognition, Christopher M. Bishop.

[2] Fitting affine and orthogonal transformations between two sets of points, Helmuth Späth, 2004

$\endgroup$
3
  • $\begingroup$ Is it possible to find a closed-form solution if there is an activation function? And will the closed form solution also provide the best approximation if the equation has no perfect solution? If w, x, and y are scalars ,for example, and the training data for (x,y) are (1,1) and (1,2), then an optimal solution (under a MSE loss function) would be w = 1.5. $\endgroup$
    – Yaoshiang
    Apr 3, 2022 at 16:50
  • 1
    $\begingroup$ It's definitely possible if the activation is invertible, like a sigmoid or a tanh. Then you can just pass the targets through the inverse activation. This doesn't work for something non-invertible like a ReLU, but you could probably figure out something specific for the ReLU. If there is no exact solution, then the pseudo-inverse will give you the least squares solution (i.e. the output closest to the target). $\endgroup$
    – Peter
    Apr 6, 2022 at 10:52
  • $\begingroup$ Tanks for the tip, @peter. I ended up implementing it. Trivial, once you remember the basic rules of linear algebra and a pseudo-inverse (which conveniently is included in tensorflow: tensorflow.org/api_docs/python/tf/linalg/pinv) stackoverflow.com/questions/71792056/…. $\endgroup$
    – Yaoshiang
    May 9, 2022 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.