Take the 2-minute tour ×
Computer Science Stack Exchange is a question and answer site for students, researchers and practitioners of computer science. It's 100% free, no registration required.

I have designed a classifier M which recognizes gestures and classifies it under any category always. A gesture is classified based on the hamming distance between the sample time series y and the training time series x. The result of the classifier are probabilistic values. There are 3 classes/categories with labels A,B,C which classifies hand gestures where there are 100 samples for each class which are to be classified (single feature and data length=100). The data are different time series (x coordinate vs time). The training set is used to assign probabilities indicating which gesture has occured how many times. So,out of 10 training samples if gesture A appeared 6 times then probability that a gesture falls under category A is

P(A)=0.6 similarly P(B)=0.3

and

P(C)=0.1

Now, I am trying to compare the performance of this classifier with Bayes classifier, K-NN, Principal component analysis (PCA) and Neural Network.

  1. On what basis,parameter and method should I do it if I consider ROC or cross validate since the features for my classifier are the probabilistic values for the ROC plot hence what shall be the features for k-nn,bayes classification and PCA?
  2. Is there a code for it which will be useful.
  3. What should be the value of k is there are 3 classes of gestures?

Please help. I am in a fix.

share|improve this question
    
Isn't PCA is Principal Component Analysis? –  om-nom-nom Mar 29 '12 at 20:24
    
Thank you for pointing it out.Corrected the same. –  user1214586 Mar 29 '12 at 20:26
    
How is this question related to your earlier question cs.stackexchange.com/questions/840/…? –  Dave Clarke Mar 29 '12 at 20:35
    
This is a shorter version of that question with significance only on how to compare the classifiers. –  user1214586 Mar 29 '12 at 20:53
    
@user1214586: Is the old one still relevant? –  Raphael Mar 30 '12 at 7:14

1 Answer 1

up vote 6 down vote accepted

There isn't an easy way to do this. In fact a recent paper by Charles Parker explains some of the problems with using ROC curve measurements (there's a friendly overview at my blog - self promotion alert!!). His paper does make some recommendations on the best ways to compare different classifiers, so you will find something useful there.

share|improve this answer
    
Then what does one do if they propose a classifier or improve an existing one?Is there no general standard way to compare? –  user1214586 Mar 30 '12 at 0:59
    
Further,in case of single feature multiple class classification,should the feature be the same for comparing?I have a hard time in this since in my example,the different algorithms have different ways of classifying. So,do I need to recognize gestures for each of the classifier which would recognize in their own algorithmic design?Kindly throw some light on this aspect please. –  user1214586 Mar 30 '12 at 1:14
    
IIRC, the Parker paper does recommend some 'best practices'. –  Suresh Mar 30 '12 at 2:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.