# Parameter sharing / weight constraints in Neural Networks

I would like to train a neural network whose parameters (alternatively, weights) are subject to linear constraints such as

$w_{i,j} = w_{i',j'}$,

where $w_{i,j}$ denotes the weight from input node $i$ to hidden node $j$ in the case of a network with a single layer. (In the case of several hidden layers, it is okay to assume that only weights stemming from the same layer can be related to each other).

It seems that a special case of this is known as parameter sharing in the context of convolutional neural networks where weights have to coincide, roughly speaking, across different layers.

Surprisingly, it seems that there is not much work / need for more general parameter constraints. Being not a machine learning expert, I am wondering whether I am using the wrong keywords in my literature research.

It would be great if someone could point me towards a tool / framework which allows for linear parameter constraints. Mathematically speaking, I would like to specify a linear subspace of the parameter space and project, at each step of the backwardpropagation, the update vector $\Delta w$ onto this very subspace.

Any help will be greatly appreciated.

I'll list two approaches, either of which should work.

One general approach would be to use projected gradient descent, which provides a way to accommodate these kinds of constraints. For specific kinds of constraints there may be a better way.

Let's recall how we train neural networks. We have an objective function $\Psi$ that computes the total loss, as a function of the weights $w$. Training a neural network amounts to solving the optimization problem "minimize $\Psi(w)$". This is typically done using (stochastic) gradient descent to find the weights $w$ that minimize $\Psi(w)$. In each iteration, you take a single gradient step to update $w$ (backpropagation is just a way to help you compute the gradient $\nabla \Psi(w)$, to help you take this step).

Now you want to add some constraints on $w$. Let $\mathcal{R}$ denote the region (linear subspace) of $w$'s that satisfy all your constraints. Now we want to solve the optimization problem "minimize $\Psi(w)$ subject to $w \in \mathcal{R}$". One standard way to solve such an optimization problem is to use projected gradient descent: in each iteration, you take a single gradient step to update $w$, then project $w$ to the nearest point in $\mathcal{R}$; and repeat. So this amounts to doing the standard backprop training procedure for a neural network, but after each weight update, projecting the weights to the nearest valid value that satisfies all the constraints. You can read more about projected gradient descent in standard resources. In your situation, it is easy to project onto a linear subspace; this is a simple matter of linear algebra. So projected gradient descent should be straightforward to apply. It also shouldn't be too hard to add the projection step, in a framework like Tensorflow. This also has the nice benefit of generalizing beyond linear constraints.

# Reparameterization

The other plausible approach would be a reparameterization trick. If you have $m$ weights, your region $\mathcal{R}$ has the form $\{w : Aw=c\}$, where $A$ is a $m \times n$ matrix and $c$ a constant $m$-vector. This can be reparametrized into the form $\{Bv\}$ where $B$ is a $n\times n-m$ matrix and $v$ ranges over all possible $n-m$-vectors. In other words, we can find a matrix $B$ such that every valid weight value $w$ can be expressed as $w=Bv$ for some $v$. (I think you can compute $B$ as the pseudoinverse of $A$.) Then, instead of solving the optimization problem "minimize $\Psi(w)$ subject to $w \in \mathcal{R}$", you can solve the optimization problem "minimize $\Psi(Bv)$", where now there are no constraints on $v$. This in turn can be done using gradient descent.

Once you've found the matrix $B$, you'll probably find the latter approach very easy to implement in frameworks like Tensorflow; you just let $v$ be the variables you are trying to solve for, compute $w$ from $v$ using $w=Bv$, and then the neural network uses these derived weights $w$, and you can ask Tensorflow to minimize the loss as usual.

• Thanks a lot for your reply. Pardon me if I was not precise in my question. Is is clear to me that mathematically I have to compute merely a projection of $\Delta w$ unto the linear space in question (compare the end of my question with the first half of your answer). What I am not clear about is whether this is already supported by some established tool or framework like MATLAB or whether I have to code it up myself. Best – user20374 Jul 16 '18 at 13:14
• @user20374, my guess is that you'll have to code it yourself. – D.W. Jul 16 '18 at 15:23
• Thanks for the great answer. How about using Lagrange multipliers for the constraints? Would it be equivalent to the projection method? I have a bunch of polynomial constraints, wonder if the (non-)convexity of the permissible region would be a problem... – Yan King Yin Jul 14 '19 at 5:25