backpropagation algorithm seems to be forcing output values to middle than extremes

Question

I have been playing around with artificial neural networks lately, specifically with the prospect of trying to replace the contrastive divergence algorithm with some type of evolutionary metaheuristic technique (i have been working with PSO) to do unsupervised pre-training, but I have come up against some issues when trying to implement the backpropagation algorithm to then fine-tune the connection weights to create a classifier.

I am using a reduced version of the MNIST dataset consisting of just the '0's and '1's to try and develop a binary classifier than is capable of distinguishing between the two different classes.

The output of my network is two visible nodes that are supposed to output [1 0] if the data is classified as a '0' and [0 1] if the data is classified as a '1'

before learning with the backpropagation algorithm i have about a 30% classification error using the quadratic error, and as i perform the supervised learning, the error gets pushed towards 12.5% but stays there as if there was an asymptote.

I thought this was showing that it was improving the classification, however when i examined the output of the network, it was just showing that with each iteration, the activations of the output nodes were moving closer to [0.5 0.5] before settling there.

before backpropagation was applied it was giving output values of [0.75 0.2] (for example) - and any attempts to train it using backpropagation just pushed those values toward [0.5 0.5] rather than amplifying the difference.

can anyone offer any insight as to why this might be happening? is this a common problem when training neural networks? or does it just sound like an implementation bug? i have implemented my backpropagation algorithm in a similar fashion to that which is found here.

Answer 1

I have run into a similar issue when trying to train a neural network with hidden layers for regression. My network simply learned to always predict the mean of all target values in the training set.

In my case, the problem was missing input normalization (my input data had values in the 100,000s) in combination with sigmoid units: The easiest way for the network to reduce the loss function (mean squared error) was to simply set the weights in the early layers to very small values and the output unit's bias to the data's mean value.

After that, the network was stuck and no amount of training seemed to change anything, since the absolute gradient values were too small.