I have been playing around with artificial neural networks lately, specifically with the prospect of trying to replace the contrastive divergence algorithm with some type of evolutionary metaheuristic technique (i have been working with PSO) to do unsupervised pre-training, but I have come up against some issues when trying to implement the backpropagation algorithm to then fine-tune the connection weights to create a classifier.
I am using a reduced version of the MNIST dataset consisting of just the '0's and '1's to try and develop a binary classifier than is capable of distinguishing between the two different classes.
The output of my network is two visible nodes that are supposed to output [1 0] if the data is classified as a '0' and [0 1] if the data is classified as a '1'
before learning with the backpropagation algorithm i have about a 30% classification error using the quadratic error, and as i perform the supervised learning, the error gets pushed towards 12.5% but stays there as if there was an asymptote.
I thought this was showing that it was improving the classification, however when i examined the output of the network, it was just showing that with each iteration, the activations of the output nodes were moving closer to [0.5 0.5] before settling there.
before backpropagation was applied it was giving output values of [0.75 0.2] (for example) - and any attempts to train it using backpropagation just pushed those values toward [0.5 0.5] rather than amplifying the difference.
can anyone offer any insight as to why this might be happening? is this a common problem when training neural networks? or does it just sound like an implementation bug? i have implemented my backpropagation algorithm in a similar fashion to that which is found here.