Which machine learning algorithms (besides SVM's) use the principle of structural risk minimization?
$\begingroup$
$\endgroup$
5
-
2$\begingroup$ What is an algo? $\endgroup$– Dave ClarkeCommented May 22, 2012 at 20:29
-
$\begingroup$ algo = algorithm ;) $\endgroup$– ClassifireCommented May 22, 2012 at 20:56
-
$\begingroup$ please use complete words. $\endgroup$– KavehCommented May 22, 2012 at 23:34
-
$\begingroup$ ok..just didn't wanna make the title too long $\endgroup$– ClassifireCommented May 23, 2012 at 9:38
-
$\begingroup$ As far as I can tell SRM is nothing but good old regularization, which is used absolutely everywhere. $\endgroup$– EmreCommented May 23, 2012 at 19:52
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
The structural risk minimization principle is a principle that is at least partly 'used' in all machine learning methods, since overfitting is often to be taken into account: reducing the complexity of the model is (supposedly and in practice) a good way to limit overfitting.
SVMs explicitly have a parameter for the complexity (the dimension of the feature space, or even the kernel function) and it's necessary because increasing the complexity is a part of the learning algorithm.
Neuronal networks also have a easy indicator of their complexity (number of 'cells') and is part of the associated learning algorithm.
Without this principle grammar inference would be both stupid and perfect grammar is the list of all possible words, so every non-trivial algorithm at least acknowledges this principle.
Decision trees have their own notion of entropy.
Clusters can be simply counted or kind of 'use' the principle intrinsically or have a fixed number of clusters and in that case you apply the principle at a higher level.
To be perfectly honest I don't really know about what happens in genetic programming but they don't have an intrinsic notion of complexity.
I don't know well Inductive logic programming but it doesn't seem to scale very well to this principle.
-
$\begingroup$ Do you know of any learning algorithm that is even more powerful and less prone to overfitting than SVM? Or maybe a technique to improve standard SVM? $\endgroup$ Commented May 23, 2012 at 10:12
-
-
$\begingroup$ Well, I'd like to use SVM in the financial markets, and there are actually quite a few papers dedicated to this topic (using SVM for stock prediction, etc...). Is there an algorithm that would be better suited for that purpose (especially since financial time-series are so "noisy")? $\endgroup$ Commented May 23, 2012 at 12:58
-
$\begingroup$ @user2278 You better use the papers. I'm not an expert. (I would not be surprised SVMs are the best for that. Also they behave well wrt. noise) $\endgroup$– jmadCommented May 23, 2012 at 13:14