I tuned some hyperparameters and got 2 best models.

These two models are almost the same "test(not train)" and "validation" errors. But model A has lower "train" error than model B.

In this case, the model with lower "train" error (model A), would show good performance on new data? (good generalization)

Or model A is more overfitting to the dataset so it would work worse in new data?



Though usually the test error is your primary goal, I can imagine some specific cases when the train error also matters. One possible example: training set has much more 'hard' inputs than the test set.

If you ever visually examined MNIST, you've probably seen that some images are puzzling even for a human, while others are much easier to tell. If one would split the training/test set differently, it's possible to see that training error would be higher than the validation or test error.

However, it's important to say that a situation like that indicates that data is probably poorly collected or split. So in this case it's better not to choose model A vs model B, but to check the data.


I don't think there is any reason to expect either one to perform better on new data. Assuming you haven't used the test set for anything other than evaluating these two models, the performance on the test set is the best estimate you have of how well it will perform on new data. Thus if they get the same performance on the test set, I don't think there's any basis for expecting one or the other to be better on new data.

  • $\begingroup$ You said, "validation set is the best estimate you have of how well it will perform on new data.". Isn't it "test set" instead of "validation set"? As I know of, validation set is used for tuning hyperparameters. What do you think? $\endgroup$ – user3595632 Sep 28 '17 at 7:28
  • $\begingroup$ @user3595632, yes, you are right. Edited. Thank you. $\endgroup$ – D.W. Sep 28 '17 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.