If I split my data properly into 75% train, 15% test, and 15% validation, and there are over 100,000 samples, is it appropriate for me to train 100s of neural networks then select only a couple based on their testing set grades for publication in a research article?

The question is for neural network training in which the random initialization of weights can drastically affect test grades. Network parameters like the number of neurons, number of layers, etc all can change how the testing grade is. These are all things that are easy to modify, rather than data preparation which can take a long time.

Do I need to resplit my data every time I train a new network? What strategy do most NN theorists use?

  • 4
    $\begingroup$ No, of course it's not appropriate to cherry-pick the best results. "Oh, look. My new drug is 100% effective because I only reported the two people I cured and not the 98 people it killed." $\endgroup$ Jun 23 '16 at 19:18
  • $\begingroup$ This was tagged as a question about neural networks. See edits above $\endgroup$ Jun 24 '16 at 5:01
  • 2
    $\begingroup$ This may be a better for for Computational Science or Cross Validated, or even Academia. $\endgroup$
    – Raphael
    Jun 24 '16 at 10:34

No. You must use the test set only once. Otherwise your result will not be representative of the true performance of your scheme. That's bad.

Instead, use the validation set for checking the performance of your algorithm on different parameters, different initialization strategies, different architectures, etc. That's what the validation set is there for. Each time you have a new idea for a different approach, you train a model on the 75% training set, then evaluate how well it performs on the 15% validation set, and use that to determine what other changes you might want to try. Repeat as many times as you like -- but never use the test set for any of this. Don't re-split the data.

After you've finished all your experimentation, and are ready to commit to a single solution, now evaluate it on your test set, and publish whatever you get. (But the one thing you absolutely must avoid is, after seeing its performance on the test set, deciding to go make other changes and then re-evaluate on the test set a second time. Don't do that!)

Alternatively, you can pool the training + validation data into a single group, and use cross-validation on that as you test each variation. Again, don't use the test data until the very end, and use it only once.

This should be well-documented in a good reference on machine learning.

  • $\begingroup$ See edits about number of neurons, etc $\endgroup$ Jun 24 '16 at 5:06
  • $\begingroup$ Training the model once can be unrepresentative as well. If there is lots of jitter in the model (which is probably bad) you'll need many independent training runs to realize this. Of course, you have to report that some training runs result in very different models than others! $\endgroup$
    – Raphael
    Jun 24 '16 at 10:36
  • $\begingroup$ @koampapapa, see edited answer. $\endgroup$
    – D.W.
    Jun 24 '16 at 15:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.