Neural Network Normalization and de-Normalisation of data

Question

I am developing a simple backprop neural network with n inputs and 1 output. I am using a sigmoid activation function. [Aforge.Net]

I have read that it is good to normalise the input and output data prior to training, which I have done using a simple linear relation (max/min mapping) to normalise between [0, 1].

My question is, what happens when upon using the NN after training, I get inputs that are larger/smaller than the max/min of the training sets (used for normalisation)? Should I be storing new max/mins and using these to re-normalise my new inputs? or do I just normalise the data with the parameters used in the training set and simply give the network inputs that may be greater than 1 or less than 0?

Answer 1

You should definitely not be storing new max/mins after training is finished since that will mess up the values for the values that are within the old max and min.

You're network might not do very well on any of these new values anyway since they are more extreme than anything it saw during training. You should scale everything using the max/min you found during training. You could cap anything more extreme than that to be either 0 or 1.

Answer 2

I only want to add a few things to Aaron answer: Experimentally, I found it to be more effective to normalize the data with its variance and mean, i.e., given feature X, (X_new X-mean(X))/var(X)^.5.

Anyway, I have to point out that normalization is NOT a necessary step for neural net training, by initializing the weights with very small values you can easily handle the non-normalized datasets, though normalization usually increase the training speed.