Computer Science Stack Exchange is a question and answer site for students, researchers and practitioners of computer science. It only takes a minute to sign up.
Sign up to join this community
I tuned some hyperparameters and got 2 best models.
These two models are almost the same "test(not train)" and "validation" errors. But model A has lower "train" error than model B.
In this case, the model with lower "train" error (model A), would show good performance on new data? (good generalization)
Or model A is more overfitting to the dataset so it would work worse in new data?
Thanks
Though usually the test error is your primary goal, I can imagine some specific cases when the train error also matters. One possible example: training set has much more 'hard' inputs than the test set.
If you ever visually examined MNIST, you've probably seen that some images are puzzling even for a human, while others are much easier to tell. If one would split the training/test set differently, it's possible to see that training error would be higher than the validation or test error.
However, it's important to say that a situation like that indicates that data is probably poorly collected or split. So in this case it's better not to choose model A vs model B, but to check the data.
I don't think there is any reason to expect either one to perform better on new data. Assuming you haven't used the test set for anything other than evaluating these two models, the performance on the test set is the best estimate you have of how well it will perform on new data. Thus if they get the same performance on the test set, I don't think there's any basis for expecting one or the other to be better on new data.
