2
$\begingroup$

I am new in the ML. I know that overfitting is memorizing the data while training. Like in Neural Network, if we make lots of layers and lots of hidden nodes, we can memorize all the data, but it can be bad because train data would not cover the whole space.

Like this, is there any way to overfit in SVM and Logistic Regression? Since they are linear algorithms, they cannot be something curvy, I guess, so I am guessing the answer would be no. But I am not sure.

Any help is appreciated.

$\endgroup$
  • $\begingroup$ As D.W. and Seb pointed out above, yes SVMs and logical regressions technically can be overfitted, though they are considered to be resistant to overfitting since they are linear, as you pointed out. I found this <a href="stats.stackexchange.com/questions/35276/…> on CrossValidated which might shed some more light on the subject! $\endgroup$ – Tyler Roesler Apr 12 '17 at 13:16
2
$\begingroup$

Yes, they can overfit too. Overfitting is especially a risk when the number of features is much larger than the number of samples in the training set.

$\endgroup$
  • $\begingroup$ I am working on a beginning level of project. While neural network results in 95 percentage of accuracy, logistic regression gives 20 percent. What would be the problem? What are the parameters of LR so that I can tune it? $\endgroup$ – Sami Şimşekli Mar 22 '17 at 18:50
  • 2
    $\begingroup$ @SamiŞimşekli, that's a different question. We ask that each question be posted separately using the 'Ask Question' button -- but make sure to show what you've tried, what your thoughts are, what research you've done, and what steps you've taken to try to diagnose the issue. Also, make sure to ask only one question per post. $\endgroup$ – D.W. Mar 22 '17 at 18:54
1
$\begingroup$

As D.W. points out, in principle every machine learning algorithm can overfit a finite data sample provided you give it enough flexibility and degrees of freedom, e.g., by adding layers or additional features.

However, different methods will be more or less prone to overfitting, and their tendency to overfit is typically studied by theoretical notions such as Rademacher complexity or the Vapnik-Chervonenkis dimension, that roughly speaking characterizes the maximal number of points that an algorithm can perfectly fit for sure.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.