Which machine learning algorithms (besides SVM's) use the principle of structural risk minimization?

  • 2
    $\begingroup$ What is an algo? $\endgroup$ May 22, 2012 at 20:29
  • $\begingroup$ algo = algorithm ;) $\endgroup$
    – Classifire
    May 22, 2012 at 20:56
  • $\begingroup$ please use complete words. $\endgroup$
    – Kaveh
    May 22, 2012 at 23:34
  • $\begingroup$ ok..just didn't wanna make the title too long $\endgroup$
    – Classifire
    May 23, 2012 at 9:38
  • $\begingroup$ As far as I can tell SRM is nothing but good old regularization, which is used absolutely everywhere. $\endgroup$
    – Emre
    May 23, 2012 at 19:52

1 Answer 1


The structural risk minimization principle is a principle that is at least partly 'used' in all machine learning methods, since overfitting is often to be taken into account: reducing the complexity of the model is (supposedly and in practice) a good way to limit overfitting.

  • SVMs explicitly have a parameter for the complexity (the dimension of the feature space, or even the kernel function) and it's necessary because increasing the complexity is a part of the learning algorithm.

  • Neuronal networks also have a easy indicator of their complexity (number of 'cells') and is part of the associated learning algorithm.

  • Without this principle grammar inference would be both stupid and perfect grammar is the list of all possible words, so every non-trivial algorithm at least acknowledges this principle.

  • Decision trees have their own notion of entropy.

  • Clusters can be simply counted or kind of 'use' the principle intrinsically or have a fixed number of clusters and in that case you apply the principle at a higher level.

To be perfectly honest I don't really know about what happens in genetic programming but they don't have an intrinsic notion of complexity.

I don't know well Inductive logic programming but it doesn't seem to scale very well to this principle.

  • $\begingroup$ Do you know of any learning algorithm that is even more powerful and less prone to overfitting than SVM? Or maybe a technique to improve standard SVM? $\endgroup$
    – Classifire
    May 23, 2012 at 10:12
  • $\begingroup$ @user2278 if by 'powerful' you mean 'efficient' then SVMs are pretty great and there is a lot of research about it and tools using it. But of course, it depends on your problem. $\endgroup$
    – jmad
    May 23, 2012 at 12:39
  • $\begingroup$ Well, I'd like to use SVM in the financial markets, and there are actually quite a few papers dedicated to this topic (using SVM for stock prediction, etc...). Is there an algorithm that would be better suited for that purpose (especially since financial time-series are so "noisy")? $\endgroup$
    – Classifire
    May 23, 2012 at 12:58
  • $\begingroup$ @user2278 You better use the papers. I'm not an expert. (I would not be surprised SVMs are the best for that. Also they behave well wrt. noise) $\endgroup$
    – jmad
    May 23, 2012 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.