I am trying to apply Linear Regression method for a dataset of 9 sample with around 50 features using python. I have tried different methodology for Linear Regression i.e Closed form OLS(Ordinary Least Squares), LR(Linear Regression), HR(Huber Regression), NNLS( Non negative least squares) and each of them gives different weights.

But I can get the intuition why HR and NNLS has different solution, but LR and Closed form OLS have the same objective function of minimizing the sum of the squares of the differences between observed value in the given sample and those predicted by a linear function of a set of features. Since the training set is singular, i had to use pseudoinverse to perform Closed form OLS.

$\beta ={(A^{T}A)}^{-1}A^Ty$

For LR i have used scikit-learn Linear Regression which uses lapack library from http://www.netlib.org/lapack/lug/node27.html to solve the least-squares problem

System of linear equations or a system of polynomial equations is referred as underdetermined if no of equations available are less than unknown parameters. Each unknown parameter can be counted as an available degree of freedom. Each equation presented can be applied as a constraint that restricts one degree of freedom. As a result an underdetermined system can have infinitely many solutions or no solution at all. Since in our case study, system is underdetermined and also is singular, there exists many solutions.

Now both pseudoinverse and Lapack library tries to finds minimum norm solution of an underdetermined system when no of sample is less than no of features. Then why the closed form and LR gives completely different solution of the same system of linear equations. Am i missing something here which can explain the behaviors of both ways. Like if the peudoinverse is computed in different ways like SVD, QR/LQ factorization, can they produce different solution for same set of equations?

  • $\begingroup$ Welcome to CS.SE! Your question is reasonable, I suspect this might be more of a question about the specific implementations found in scikit-learn than any fundamental concepts (and if so, it's probably not on-topic here). Does scikit-learn use regularization by default for either of those linear regression solvers? $\endgroup$ – D.W. Sep 28 '17 at 0:04
  • $\begingroup$ Yes, i agree the question is not completely theoretical.But I wanted to clear my main concern which is purely theoretical i think i should have included in my post to be precise. That is in machine learning paradigm, if closed form OLS and linear regression are same thing as the closed form is derived from the same objective function as in LR or they might depend on the implementation? scikit-learn doesn't use regularization by default for either of those linear regression solvers. $\endgroup$ – Farzana Yusuf Sep 28 '17 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.