Why do neural networks seem to perform better with restrictions placed on their topology?

Question

Fully connected (at least layer to layer with more than 2 hidden layers) backprop networks are universal learners. Unfortunately, they are often slow to learn and tend to over-fit or have awkward generalizations.

From fooling around with these networks, I have observed that pruning some of the edges (so that their weight is zero and impossible to change) tends to make the networks learn faster and generalize better. Is there a reason for this? Is it only because of a decrease in the dimensionality of the weights search space, or is there a more subtle reason?

Also, is the better generalization an artifact of the 'natural' problems I am looking at?

Answer 1

Fewer nodes/edges (or edges with fixed weights) means that there are fewer parameters whose values need to be found, and this typically reduces the time to learn. Also, when there are fewer parameters, the space that can be expressed by the neural network has fewer dimensions, so the neural network can only express more general models. It is thus is less capable of over-fitting the data, and hence the models will seem more general.

Answer 2

By pruning edges you've reduced the search space for the training algorithm, which will have an immediate payoff in time performance. You've also introduced constraints on the functions the network can model. The constraints may force your model to find a more general solution since the more accurate one is unreachable. A common technique for training neural networks is using a gradient descent technique. Another consequence of the pruning may be that you've eliminated some local minima in the parameter landscape that again allows the training algorithm to find a better solution.

I would not be surprised if your better generalization is related to the problems you're looking at. I've enjoyed success with neural networks where the underlying model has a continuous structure, while cases where there are discontinuities things didn't work so well. Keep in mind also that neural network performance is often intimately related to how you structure you input and output.