I am aware of three approaches to prevent over-fitting of data when trying to model it on a neural net.

The first two approaches I know suggest to train on more data and employ bootstrap aggregating. The third approach is to use a model that has the right capacity, one that has enough to fit the true regularities but not the weaker/dubious regularities (the noise). However, I don't quite understand this 2nd approach. How do we create a distinction and choose a model that detects this? Can someone expand on the second approach? I am aware of ways to limit the capacity such as early stopping(which is self explanatory), penalty for peculiar weights, etc. But an explanation on why these specific methods limit the capacity to fit the right data but not the wrong one would be helpful.

  • $\begingroup$ You mentioned :"second approaches(2) suggest to train on more data and employ bootstrap aggregating". Can you give more info about this approach? Did you read it from a research paper/journal etc? $\endgroup$ Commented Oct 22, 2016 at 10:46
  • $\begingroup$ "this 2nd approach" - what do you mean by this? are you referring to bootstrap aggregating? If so, what research have you done? What have you read so far? Have you read en.wikipedia.org/wiki/Bootstrap_aggregating? What gave you the idea that bootstrap aggregating reduces over-fitting? Or, are you asking about reducing capacity? If so, it's confusing to call it "the third approach" and then refer to it as "this 2nd approach". What do you mean by "create a distinction and choose a model that detects this"? $\endgroup$
    – D.W.
    Commented Oct 22, 2016 at 10:52
  • $\begingroup$ Also, what's your specific question? "Can someone expand on the second approach?" is vague and open-ended. We seek narrowly focused technical questions that can be answered in a few paragraphs. What specifically are you confused about? Also, it sounds like you have in mind a whole list of approaches that fall under that category and want us to explain something about all of them. That seems too broad; it amounts to multiple different questions. Finally, there's lots written about overfitting; what have you read? $\endgroup$
    – D.W.
    Commented Oct 22, 2016 at 10:55
  • $\begingroup$ stats.stackexchange.com/a/187700/2921 $\endgroup$
    – D.W.
    Commented Oct 22, 2016 at 11:10
  • $\begingroup$ I think OP is referring to "The third approach", i.e. "choosing a model with the right capacity". $\endgroup$
    – Ariel
    Commented Oct 22, 2016 at 11:42

1 Answer 1


When choosing a specific model, to which you will try to fit your data into, you must know something about the problem. There is no way around it, machine learning cant do this magic for us, we have to know something.

First, convince yourself why complicated models can cause overfitting. Suppose I'm trying to learn some real valued function $f$, and I'm given $n$ samples of the form $S=(x_1,y_1),...,(x_n,y_n)$. If i choose to model my data by some polynomial (of arbitrary degree) then for any set of samples $S$, i can find a function in my model with zero training error, although it is likely to have high generalization error (a degree $n$ polynomial passing through all points in $S$ can always be found, regardless of any structure of the points in $S$). Perhaps using polynomials of degree$ \le 3$ will be good enough to model my data (it doesn't suffer from the previous problem). Why 3? I don't know, this depends on the specific types of functions I'm trying to learn.

A more concrete example, suppose you want to learn what is the appropriate dosage of some medicine, and you are given samples of the form $(x_1,y_1),...,(x_n,y_n)$ where $x_i\in\mathbb{R}$ denotes the dosage, and $y_i\in\left\{\text{'good'},\text{'bad'}\right\}$ tells you whether the dosage $x_i$ worked (it can be too high or too low). Allowing your algorithm to choose some arbitrary function can result in strange outputs. For example if you received the samples $(1,\text{'bad'}),(2,\text{'good'}),(3,\text{'good'})$ then a function $f:\mathbb{R}\rightarrow\left\{-1,1\right\}$ which satisfies $f(1)=-1,f(2)=f(3)=1$, but is random everywhere else, is consistent with the samples. Such $f$ will probably have no indication on the true effect of the medicine. Now, I'm no doctor, but perhaps it is worth considering functions of the form:

$I_{a,b}=\begin{cases} +1, & x\in [a,b] \\ -1, & \text{otherwise} \end{cases}$.

When choosing to model my data using such functions (segments), i use my knowledge on how medicines work (there is some appropriate dosage interval, while large deviations to any side could be dangerous).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.