How to reconstruct the image from a neural network output?

Question

I am trying to use the genetic algorithm to optimise a multi-layered neural network for image classification (i am using a subset of the MNIST handwritten digit data set as my initial dataset, but eventually would like to make it more general).

my neural network is represented by connection weight matrices with inputs as the rows an outputs as the columns (so element (3, 4) is the weight of the connection between input 3 and output 4, etc) and have an extra column and row for the biases of the inputs/outputs

the system seems to work well, it takes an input vector, multiplies it by the first layer matrix, uses the sigmoid function to determine whether that neuron will fire or not, then uses the output vector of that as the input to multiply by the 2nd layer and so on to the final layer, and i am getting the results that i expect with test data that i know the expected output for.

i have an idea of how i will implement the genetic algorithm to search for the best solutions, but my problem is that im having a little bit of trouble understanding how neural networks can be used to reconstruct images so i can test for the error between the input image and the reconstruction to test each solution for fitness.

i understand that i need to first encode the image into a vector of pixels, and that that input needs to be passed through the network to the end layer, but i am not sure what that end data is supposed to represent..

the dataset i am using are images of 28*28 pixels, so the structure of my network is as follows (inspired by the Hinton paper here):

484 -> 1000 -> 500 -> 250 -> 30

so the output of the whole thing will be a vector 31 (30 neurons +1 for the bias column/row) elements long

how should i use this vector to reconstruct the image?

i read somewhere that to decode information from a neural network you need to multiply the output by the transpose of the transformation matrix, but that doesnt really make sense as (for example):

T = [ 1  2 ;
      3  4 ]

A = [ 5  6 ]

A * T = [ 23  34 ] = B

B * T' = [ 91  205 ] != A

clearly this is not a very good reconstruction of the original data..

can someone give an explanation as to how neural nets can be used to reconstruct data, and if possible, point me to some good resources on the subject?

Answer 1

You can't necessarily use the output of a neural network to reconstruct the image, like you want. The output signals indicate how the neural network has classified the input image. Beyond that, they don't necessarily have any interpretation as a representation of the original image. There isn't necessarily any way to do what you want (to reconstruct the image given the output of the neural network), as this kind of neural network inherently throws away information.

Are you familiar with the notion of training "autoencoders"? One way to train multi-layer neural networks is by building a series of auto-encoders: e.g., to construct the weights for the first layer, you train a 2-layer autoencoder, then throw away the second layer of the autoencoder and use the first layer of the autoencoder as the first layer of your neural network. You repeatedly do this for each layer of your network. Autoencoders do reconstruct the original input, so if you're training each layer of the neural network in this way, then you automatically have a way to reconstruct (an approximation to) the original input from the signals at any layer -- if you saved the entire autoencoder at each stage. However, if your real end goal is to understand why an image was classified a certain way or debug your neural network, I expect this probably won't be helpful for that.

I'm not 100% sure what your real goal is. You mention "test each solution for fitness", but I'm not sure what you meant by that. However, my guess is that your best bet will be to figure out what your real goal is, and then ask about how to achieve that goal (which might not be by trying to reconstruct the original image).