# Over-fitting Always Occurs?

i get stuck in one sentence in machine learning.

i read tom Mitchel book on ML, and some other materials.

if we have small training set, always over-fit can occurs? or is likely to occurs?

i read lots of material but i'm not sure that which of my fact will be correct.

any solution or idea?

-

Roughly speaking, over-fitting typically occurs when the ratio $\frac{\text{complexity of the model}}{\text{training set size}}$ is too high. Think of over-fitting as a situation where your model learn the training data by heart instead of learning the big pictures which prevent it from being able to generalized to the test data: this happens when the model is too complex with respect to the size of the training data, that is to say when the size of the training data is to small in comparison with the model complexity.

Examples:

• if your data is in two dimensions, you have 10000 points in the training set and the model is a line, you are likely to under-fit.
• if your data is in two dimensions, you have 10 points in the training set and the model is 100-degree polynomial, you are likely to over-fit.

From a theoretical standpoint, the amount of data you need to properly train your model is a crucial yet far-to-be-answered question in machine learning. One such approach to answer this question is the VC dimension. Another is the bias-variance tradeoff.

From an empirical standpoint, people typically plot the training error and the test error on the same plot and make sure that they don't reduce the training error at the expense of the test error:

I would advise to watch Coursera' Machine Learning course, section "10: Advice for applying Machine Learning".

-

Overfitting occurs whenever there is noise, because the definition of overfitting is (more or less) fitting a model to the noise, instead of to the signal.

Since noise always exists in real data, overfitting always happens when working with real data.

-
I'm not sure I agree with this opinion. –  Yuval Filmus Jun 18 '14 at 7:31
Not true. You can have noiseless data, and still overfit. Overfit is the fact that you have a high variance in your measurements of the classifier performance –  juampa Jun 18 '14 at 23:03
@YuvalFilmus: Well unless you can mention a reason for it, that's not a very interesting comment! –  Mehrdad Jun 18 '14 at 23:08
@juampa: "Overfit is the fact that you have a high variance in your measurements of the classifier performance"... there are at least 2 reasons why your definition doesn't make any sense. (1) How in the world do you obtain nonzero variance in measurements without noise?! Wouldn't you get back the same measurements over and over again?? (2) Overfitting is not a property of your measurements of the classifier's performance, it's a property of the classifier's performance itself. If you measure it incorrectly, that doesn't mean the classifier has actually overfit!! –  Mehrdad Jun 18 '14 at 23:15

Overfit is usually characterized by high variance in the estimation of the classifier performance. When having very little data one tends to fall in this regime. It is in this case where techniques like bootstrapping become useful.

The idea is that your measurements are inherently unstable due to the lack of data, because the system to be solved is undetermined. Hence you observe high variance.

You need to introduce some smoothing that gets you closer to the task data. Bootstrapping and generating new samples by adding small amount of noise usually help in this regime.

-
you means always with small training data overfit not happened ? –  Ebraham Jun 18 '14 at 23:21