# Methods to prevent overfitting

I am aware of three approaches to prevent over-fitting of data when trying to model it on a neural net.

The first two approaches I know suggest to train on more data and employ bootstrap aggregating. The third approach is to use a model that has the right capacity, one that has enough to fit the true regularities but not the weaker/dubious regularities (the noise). However, I don't quite understand this 2nd approach. How do we create a distinction and choose a model that detects this? Can someone expand on the second approach? I am aware of ways to limit the capacity such as early stopping(which is self explanatory), penalty for peculiar weights, etc. But an explanation on why these specific methods limit the capacity to fit the right data but not the wrong one would be helpful.

• You mentioned :"second approaches(2) suggest to train on more data and employ bootstrap aggregating". Can you give more info about this approach? Did you read it from a research paper/journal etc? – Alwyn Mathew Oct 22 '16 at 10:46
• "this 2nd approach" - what do you mean by this? are you referring to bootstrap aggregating? If so, what research have you done? What have you read so far? Have you read en.wikipedia.org/wiki/Bootstrap_aggregating? What gave you the idea that bootstrap aggregating reduces over-fitting? Or, are you asking about reducing capacity? If so, it's confusing to call it "the third approach" and then refer to it as "this 2nd approach". What do you mean by "create a distinction and choose a model that detects this"? – D.W. Oct 22 '16 at 10:52
• Also, what's your specific question? "Can someone expand on the second approach?" is vague and open-ended. We seek narrowly focused technical questions that can be answered in a few paragraphs. What specifically are you confused about? Also, it sounds like you have in mind a whole list of approaches that fall under that category and want us to explain something about all of them. That seems too broad; it amounts to multiple different questions. Finally, there's lots written about overfitting; what have you read? – D.W. Oct 22 '16 at 10:55
• stats.stackexchange.com/a/187700/2921 – D.W. Oct 22 '16 at 11:10
• I think OP is referring to "The third approach", i.e. "choosing a model with the right capacity". – Ariel Oct 22 '16 at 11:42

First, convince yourself why complicated models can cause overfitting. Suppose I'm trying to learn some real valued function $f$, and I'm given $n$ samples of the form $S=(x_1,y_1),...,(x_n,y_n)$. If i choose to model my data by some polynomial (of arbitrary degree) then for any set of samples $S$, i can find a function in my model with zero training error, although it is likely to have high generalization error (a degree $n$ polynomial passing through all points in $S$ can always be found, regardless of any structure of the points in $S$). Perhaps using polynomials of degree$\le 3$ will be good enough to model my data (it doesn't suffer from the previous problem). Why 3? I don't know, this depends on the specific types of functions I'm trying to learn.
A more concrete example, suppose you want to learn what is the appropriate dosage of some medicine, and you are given samples of the form $(x_1,y_1),...,(x_n,y_n)$ where $x_i\in\mathbb{R}$ denotes the dosage, and $y_i\in\left\{\text{'good'},\text{'bad'}\right\}$ tells you whether the dosage $x_i$ worked (it can be too high or too low). Allowing your algorithm to choose some arbitrary function can result in strange outputs. For example if you received the samples $(1,\text{'bad'}),(2,\text{'good'}),(3,\text{'good'})$ then a function $f:\mathbb{R}\rightarrow\left\{-1,1\right\}$ which satisfies $f(1)=-1,f(2)=f(3)=1$, but is random everywhere else, is consistent with the samples. Such $f$ will probably have no indication on the true effect of the medicine. Now, I'm no doctor, but perhaps it is worth considering functions of the form:
$I_{a,b}=\begin{cases} +1, & x\in [a,b] \\ -1, & \text{otherwise} \end{cases}$.