I am implementing an autoencoder neural network in matlab, the weights of which are being optimised by a genetic algorithm.
At the moment I am working on the first layer, trying to get an improved reconstruction from it by using the genetic algorithm.
Using the genetic algorithm, I am able to get about a 30% improvement in fitness before the population converges, compared with the random data that it starts with; which is a reasonable improvement I thought, however it doesn't seem to make a difference with the actual reconstruction.
I will illustrate with a screenshot of my data:
At the top is shown the original data (using MNIST data), followed by the reconstruction of the data by the fittest, median and least fit individuals (at this stage the population had converged, so there is no difference between the three)
Below that is a plot of the fitness function on the y-axis (it is a minimising problem) against epochs on the x-axis for those three data points, as you can see, the fittest individual starts with a value of about 120 and finishes around 80 without improving much on the reconstruction.
My problem is with my fitness function I think. I am using the sum of the squared differences between the (normalised from 0 to 1) pixel intensity values of the original and reconstructed data - but the more I think about it, the more it seems that it will probably favour offspring that reconstruct a purely black image, as that will have a pretty low value using that heuristic; and indeed it appears to tend toward that as I watch the reconstructions evolve.
So my question is: what can i use as an additional fitness metric to try and make my algorithm favour offspring that produce less noisy reconstructions?
some I have thought of are:
- proportion of light pixels to dark pixels;
- difference between histograms of original and reconstruction;
- putting additional weight for very light pixels or very dark ones.
And that's about it. Would any of those be useful? can someone suggest anything better? I don't have much of a background in image processing, so any help would be greatly appreciated.
EDIT: In response to deong's replies;
My network goes 784 -> 500 -> 250 -> 100, training each layer one at a time as in the geoff hinton code before moving on to the next layer.
I have tried population sizes of 10-200, with many in between, i am using tournament selection and have tried k values of 2 to around 40, with a mutation rate of 0.01 - which selects a random block of 10000 values and replaces it with another 10000 random values.
As far as crossover goes; i have tried 1, 2, 3 and 4 point, uniform crossover with blocks from 100 values to 100000 values - and many variations of all of those methods, but none really seemed to make any difference, it just changes the speed at which it converges to the same thing.
I have also tried using particle swarm optimisation, and that gives me pretty much the same results - the overall mean squared error goes down over time, but the reconstruction just tends to a uniform black image, which makes sense that the mean squared error would be lower for an all black image than a random noise image because the original image is predominantly black itself.
do you have any suggestions as to what i should try next? I have started working more on the PSO algorithm lately, as it is much simpler to implement than the GA and has less variables that i need to try.