I have designed a classifier M which recognizes gestures and classifies it under any category always. A gesture is classified based on the hamming distance between the sample time series y and the training time series x. The result of the classifier are probabilistic values. There are 3 classes/categories with labels A,B,C which classifies hand gestures where there are 100 samples for each class which are to be classified (single feature and data length=100). The data are different time series (x coordinate vs time). The training set is used to assign probabilities indicating which gesture has occured how many times. So,out of 10 training samples if gesture A appeared 6 times then probability that a gesture falls under category A is

P(A)=0.6 similarly P(B)=0.3



Now, I am trying to compare the performance of this classifier with Bayes classifier, K-NN, Principal component analysis (PCA) and Neural Network.

  1. On what basis,parameter and method should I do it if I consider ROC or cross validate since the features for my classifier are the probabilistic values for the ROC plot hence what shall be the features for k-nn,bayes classification and PCA?
  2. Is there a code for it which will be useful.
  3. What should be the value of k is there are 3 classes of gestures?

Please help. I am in a fix.

  • $\begingroup$ Isn't PCA is Principal Component Analysis? $\endgroup$
    – om-nom-nom
    Mar 29, 2012 at 20:24
  • $\begingroup$ How is this question related to your earlier question cs.stackexchange.com/questions/840/…? $\endgroup$ Mar 29, 2012 at 20:35
  • $\begingroup$ This is a shorter version of that question with significance only on how to compare the classifiers. $\endgroup$ Mar 29, 2012 at 20:53
  • $\begingroup$ @user1214586: Is the old one still relevant? $\endgroup$
    – Raphael
    Mar 30, 2012 at 7:14
  • $\begingroup$ @Raphael: Yes,the old Question is relevant since the part pertaining to features selection and usage of k-nn and matlab code together with how to go about with other classifiers is not answered/unclear. Thank you and would be obliged for a solution. $\endgroup$ Mar 30, 2012 at 16:58

2 Answers 2


There isn't an easy way to do this. In fact a recent paper by Charles Parker explains some of the problems with using ROC curve measurements (there's a friendly overview at my blog - self promotion alert!!). His paper does make some recommendations on the best ways to compare different classifiers, so you will find something useful there.

  • $\begingroup$ Then what does one do if they propose a classifier or improve an existing one?Is there no general standard way to compare? $\endgroup$ Mar 30, 2012 at 0:59
  • $\begingroup$ Further,in case of single feature multiple class classification,should the feature be the same for comparing?I have a hard time in this since in my example,the different algorithms have different ways of classifying. So,do I need to recognize gestures for each of the classifier which would recognize in their own algorithmic design?Kindly throw some light on this aspect please. $\endgroup$ Mar 30, 2012 at 1:14
  • $\begingroup$ IIRC, the Parker paper does recommend some 'best practices'. $\endgroup$
    – Suresh
    Mar 30, 2012 at 2:10
  1. The general way to compare classifiers is to compare them using ROC. For a fixed range of false alarm rates you check which classifier has achieved the highest accuracy. If you want a single measure then you can use Area Under ROC (AUC). There are many other measures too, look them up. For a fair comparison you need to compare all classifiers on an even footing, this means the same features and the same training conditions.

  2. The value of k is something that you will need to find out by trying different values. Pick the value that performs best on your validation set, but only report results on your test set, which you haven't used during parameter tuning.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.