9
$\begingroup$

Which machine learning algorithms (besides SVM's) use the principle of structural risk minimization?

$\endgroup$
  • 2
    $\begingroup$ What is an algo? $\endgroup$ – Dave Clarke May 22 '12 at 20:29
  • $\begingroup$ algo = algorithm ;) $\endgroup$ – Classifire May 22 '12 at 20:56
  • $\begingroup$ please use complete words. $\endgroup$ – Kaveh May 22 '12 at 23:34
  • $\begingroup$ ok..just didn't wanna make the title too long $\endgroup$ – Classifire May 23 '12 at 9:38
  • $\begingroup$ As far as I can tell SRM is nothing but good old regularization, which is used absolutely everywhere. $\endgroup$ – Emre May 23 '12 at 19:52
8
$\begingroup$

The structural risk minimization principle is a principle that is at least partly 'used' in all machine learning methods, since overfitting is often to be taken into account: reducing the complexity of the model is (supposedly and in practice) a good way to limit overfitting.

  • SVMs explicitly have a parameter for the complexity (the dimension of the feature space, or even the kernel function) and it's necessary because increasing the complexity is a part of the learning algorithm.

  • Neuronal networks also have a easy indicator of their complexity (number of 'cells') and is part of the associated learning algorithm.

  • Without this principle grammar inference would be both stupid and perfect grammar is the list of all possible words, so every non-trivial algorithm at least acknowledges this principle.

  • Decision trees have their own notion of entropy.

  • Clusters can be simply counted or kind of 'use' the principle intrinsically or have a fixed number of clusters and in that case you apply the principle at a higher level.

To be perfectly honest I don't really know about what happens in genetic programming but they don't have an intrinsic notion of complexity.

I don't know well Inductive logic programming but it doesn't seem to scale very well to this principle.

$\endgroup$
  • $\begingroup$ Do you know of any learning algorithm that is even more powerful and less prone to overfitting than SVM? Or maybe a technique to improve standard SVM? $\endgroup$ – Classifire May 23 '12 at 10:12
  • $\begingroup$ @user2278 if by 'powerful' you mean 'efficient' then SVMs are pretty great and there is a lot of research about it and tools using it. But of course, it depends on your problem. $\endgroup$ – jmad May 23 '12 at 12:39
  • $\begingroup$ Well, I'd like to use SVM in the financial markets, and there are actually quite a few papers dedicated to this topic (using SVM for stock prediction, etc...). Is there an algorithm that would be better suited for that purpose (especially since financial time-series are so "noisy")? $\endgroup$ – Classifire May 23 '12 at 12:58
  • $\begingroup$ @user2278 You better use the papers. I'm not an expert. (I would not be surprised SVMs are the best for that. Also they behave well wrt. noise) $\endgroup$ – jmad May 23 '12 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.