1
$\begingroup$

As nicely described in this article, backpropagation for multi-layer perceptrons defines the error term for a neuron in terms of the partial derivative of the weights.

It's traditional to represent the weights between neutrons in matrix form: $w_{ij}$ representing the weight between neuroma $i$ and $j$. Since a weight can be considered as being as constant multiplier of any input, so a possible generalisation is instead to replace the $w_{ij}$ with some univariate function $f_{ij}$.

Since. a 3 layer MLP is already a universal approximator for continuous functions, this generalisation won't be more expressive, but the hypothesis is that it might converge faster for some problems.

My question then concerns the nature of the learning algorithm for this extended formulation:

How might it be possible to define backpropagation if the weight matrix is replaced by a matrix of functions? The functions can be assumed to be differentiable as required.

|cite|improve this question
$\endgroup$
1
$\begingroup$

The problem isn't well-defined as you've stated it. Backpropagation involves updating the weights, i.e., finding weights that cause the neural network to behave well. If you have no weights, then there's nothing to update.

For these reasons, the standard approach to building neural networks involves both weights and univariate functions. In the standard perceptron architecture, there are weights $w_{ij}$ along each edge (from neuron $i$ to neuron $j$), and a univariate function $f_j$ applied to the output of each neuron (neuron $j$). So standard artificial neural networks already incorporate the univariate function. The univariate function $f_j$ is called an "activation function".

Standard backpropagation does work with arbitrary activation functions. You don't need to modify or generalize it: it already deals with them. In particular, backprop can be used with an arbitrary activation function, as long as you know its derivative. So, I suggest you take another look at how standard backprop works.

Backprop is basically just (a) computation of the partial derivatives (the partial derivative of some objective function, with respect to the weights), computed using the chain rule, plus (b) gradient descent. Introducing arbitrary univariate functions affects only part (a), and the chain rule from calculus tells you how to calculate the partial derivatives.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ "If you have no weights, there's nothing to update" - well, this is the heart of the question: what )if it exists) is the corresponding notion of updating a function? I can see how this might be done in implementation terms (represent a function as an expression tree, and add some delta value to it as required) but what I'm looking for is whether this is mathematically meaningful. $\endgroup$ – NietzscheanAI Dec 14 '15 at 23:16
  • $\begingroup$ Thanks for the detailed explanation. I'm aware of generalised NNs with arbitrary diffable activation functions - it's specifically the generalisation of the weights that I'm interested in. $\endgroup$ – NietzscheanAI Dec 14 '15 at 23:20
  • $\begingroup$ @user217281728, with standard NN's, the weights are the parameters we're trying to learn. It sounds like you want to somehow make the functions be the things we're trying to learn. However, there are too many possible univariate functions, so I suspect you'd immediately overfit. To make it useful I suspect you'd need to identify a class of functions and try to choose one function from that class. The natural way to do that is to have your function depend on an additional parameter, which is to be learned. In that case, backprop could be used by taking partial derivatives. $\endgroup$ – D.W.♦ Dec 14 '15 at 23:38
  • $\begingroup$ Anyway, to say anything more specific, you'd probably have to articulate a specific proposal of exactly what architecture you have in mind for the network, what the parameters are (what you're trying to vary), and what restrictions you put on the functions (e.g., are they required to come from some class? be continuous? monotonic? something else?). But if it's all differentiable, then you should still be able to do backprop, simply by using the chain rule to compute partial derivatives. $\endgroup$ – D.W.♦ Dec 14 '15 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.