Training a model based on polynomial regression and then putting that into a linear regression, vs just using the linear regression - Computer Science Stack Exchange most recent 30 from cs.stackexchange.com 2019-09-22T08:26:09Z https://cs.stackexchange.com/feeds/question/63271 https://creativecommons.org/licenses/by-sa/4.0/rdf https://cs.stackexchange.com/q/63271 1 Training a model based on polynomial regression and then putting that into a linear regression, vs just using the linear regression Nathaniel Ostrer https://cs.stackexchange.com/users/58055 2016-09-08T15:00:40Z 2016-09-08T16:30:46Z <p>I'm doing some machine learning work in python with sklearn. Basically the problem is trying to predict how long an event will take based on 4 correlated pieces of data. What I've been doing is I have four polynomial regression trained on each piece of data respectively. Then I take the output of these polynomial regressions and train a linear regression based on that. Testing this against historically data has show that the two step process has about 10% less sum-squared-error than just using the linear regression. Should I not be doing it this way? What is the theory behind just using a one step process vs. using both steps?</p> https://cs.stackexchange.com/questions/63271/-/63274#63274 1 Answer by D.W. for Training a model based on polynomial regression and then putting that into a linear regression, vs just using the linear regression D.W. https://cs.stackexchange.com/users/755 2016-09-08T16:30:46Z 2016-09-08T16:30:46Z <p>There's no one answer about how you should/shouldn't do it. There are two perspectives out there:</p> <p><strong>If it works, go for it.</strong> One perspective is: if it gives good results, that's all that matters. So, if you get more accurate regression, great for you. Try a bunch of approaches and pick whatever works best (taking care to avoid overfitting).</p> <p><strong>Model the underlying phenomenom.</strong> The other perspective is: choose a model that represents how the underlying physical process works. So, from this perspective, we can't say what you should do, since you haven't told us how the underlying process works; we have no basis for evaluating whether that's a reasonable way to model reality.</p> <p>There are many possible models one could imagine that map 4 continuous features $x_1,x_2,x_3,x_4$ to a continuous output $y$. You could use linear regression:</p> <p>$$y = a_1 x_1 + a_2 x_2 + a_3 x_3 + a_4 x_4 + b$$</p> <p>Or you could use polynomial regression:</p> <p>$$y = p(x_1,x_2,x_3,x_4)$$</p> <p>where $p(\cdot)$ is some multivariate polynomial in 4 variables with degree at most $d$ (for some $d$ you choose). Or you could use what you're doing:</p> <p>$$y = a_1 p_1(x_1) + a_2 p_2(x_2) + a_3 p_3(x_3) + a_4 p_4(x_4) + b$$</p> <p>where $p_1(\cdot),\dots,p_4(\cdot)$ are univariate polynomials with degree at most $d$.</p> <p>Or you could fit some other nonlinear function (see <a href="https://en.wikipedia.org/wiki/Nonlinear_regression" rel="nofollow">nonlinear regression</a>), or <a href="https://en.wikipedia.org/wiki/Segmented_regression" rel="nofollow">fit a piecewise linear function</a>, or use cubic splines, or use kernel smoothing or LOESS regression (see <a href="https://en.wikipedia.org/wiki/Local_regression" rel="nofollow">local regression</a>). There are so many options. We can't tell you which to use from the information you've provided.</p> <p><strong>Advice.</strong> Usually, a good rule of thumb is that if you understand the underlying process well, use a model that matches the underlying reality. If you don't know the underlying process, try multiple approaches and see which works best (using cross-validation to avoid overfitting).</p>