Best practices for normalizing up training, validation, and test sets

Question

I was reading up on how to normalize my training, validation, and test sets for a neural network, when I read this snippet:

An important point to make about the preprocessing is that any preprocessing statistics (e.g. the data mean) must only be computed on the training data, and then applied to the validation / test data. E.g. computing the mean and subtracting it from every image across the entire dataset and then splitting the data into train/val/test splits would be a mistake. Instead, the mean must be computed only over the training data and then subtracted equally from all splits (train/val/test).

(source: http://cs231n.github.io/neural-networks-2/)

Does this mean the following?

1. Split my training set T into training set T1 & validation set V1
2. Find the mean/var of T1, mean_T1, var_T1
3. Normalize T1, V1, and my testing set with mean_T1, var_T1.
4. Train & test accordingly...

Thanks...

Answer 1

Yes, that's what it means. Basically, mean_T1 and var_T1 become part of the model that you're learning. So, same as you'd apply machine learning to the training set to learn a model based on the training set, you'll compute the mean and variance based on the training set.

Answer 2

"Must" is not a compelling scientific argument; producing a link to a document that also uses "must" as argument does not make the argument stronger unless mathematical proof or, at least, an illustrative counterexample are provided.

Using training dataset statistics to normalize validation/test data allows for information leak from train to validation and to the test data which may lead to optimistic fit/validation errors.

Taking this practice further to the new (unobserved) data will imply unchecked membership of the new data to the training population (used to built the model).

I think this practice is incorrect as it forces the stationarity assumption according to which new data is a member of the training population, by this proving the stationary of the process (circular reasoning).

Stationarity violation: when underlying processes (instantiated by training data and described by the model) have shifted, the model becomes obsolete and in need of re-training, and the new data no longer belongs to the training population.

Using the old model to predict on these new data results in high prediction errors.

A correct procedure would require the stationarity assumption be checked with each prediction by testing the membership of new data to the training population. A powerful test would require that no information from training to validation to test data is leaked through normalization or otherwise.

As a matter of fact, this happens with models not using normalized data.