I'm doing some machine learning work in python with sklearn. Basically the problem is trying to predict how long an event will take based on 4 correlated pieces of data. What I've been doing is I have four polynomial regression trained on each piece of data respectively. Then I take the output of these polynomial regressions and train a linear regression based on that. Testing this against historically data has show that the two step process has about 10% less sum-squared-error than just using the linear regression. Should I not be doing it this way? What is the theory behind just using a one step process vs. using both steps?

  • $\begingroup$ I agree with D.W.'s answer in that you should use what works, however I am inclined to disagree with your proposed two-step solution. The main reason being that it is likely just overfitting your data. When you say you take the "output" do you mean the predicted values? And then you refit a joint model using the predicted values against $y$? $\endgroup$ Sep 9, 2016 at 17:40

1 Answer 1


There's no one answer about how you should/shouldn't do it. There are two perspectives out there:

If it works, go for it. One perspective is: if it gives good results, that's all that matters. So, if you get more accurate regression, great for you. Try a bunch of approaches and pick whatever works best (taking care to avoid overfitting).

Model the underlying phenomenom. The other perspective is: choose a model that represents how the underlying physical process works. So, from this perspective, we can't say what you should do, since you haven't told us how the underlying process works; we have no basis for evaluating whether that's a reasonable way to model reality.

There are many possible models one could imagine that map 4 continuous features $x_1,x_2,x_3,x_4$ to a continuous output $y$. You could use linear regression:

$$y = a_1 x_1 + a_2 x_2 + a_3 x_3 + a_4 x_4 + b$$

Or you could use polynomial regression:

$$y = p(x_1,x_2,x_3,x_4)$$

where $p(\cdot)$ is some multivariate polynomial in 4 variables with degree at most $d$ (for some $d$ you choose). Or you could use what you're doing:

$$y = a_1 p_1(x_1) + a_2 p_2(x_2) + a_3 p_3(x_3) + a_4 p_4(x_4) + b$$

where $p_1(\cdot),\dots,p_4(\cdot)$ are univariate polynomials with degree at most $d$.

Or you could fit some other nonlinear function (see nonlinear regression), or fit a piecewise linear function, or use cubic splines, or use kernel smoothing or LOESS regression (see local regression). There are so many options. We can't tell you which to use from the information you've provided.

Advice. Usually, a good rule of thumb is that if you understand the underlying process well, use a model that matches the underlying reality. If you don't know the underlying process, try multiple approaches and see which works best (using cross-validation to avoid overfitting).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.