2
$\begingroup$

I recently did work in the area of machine learning for one of my jobs and was able to build a classifier which evaluated to an F score of 85%. I also have access to correctly and incorrectly classified instances. My boss would like to make sure my classifier is good and and test it for statistical significance. I'm not quite sure what he means by this. I have taken a course in statistics quite some time ago where I did hypothesis testing and stuff with confidence intervals. Do you think that's what he means or does he want me to compare it with other algorithms? I don't want to say what kind of data set I am working with but it's comparable to something such as predicting spam or ham for email messages.

Any help or guidance on this question would be greatly appreciated. I am pretty new to the area of computer science research. But am I right in saying that statistics are used quite a lot to evaluate these types of things?

Thank you!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

You can take a look at McNemar's Test or a Bayesian Testing. If you are comparing two or more classifiers, you can perform a to evaluate their performances.

Evaluating learning is actually a pretty wide research topic. A good book to refer to would be Japkowicz-Shah. This is a really nice book on evaluating classifiers and performances. To get some idea about the topic, take a look at this ICMLA talk.

If this is a one time thing, get done with McNemar or Bayesian test, however if you continue to work in the field, do see the book.

P.S. ask you boss if he means any real stat tests (like those mentioned above). Once I was working on a classifier, my professor asked me to evaluate its performance, after a long period of pain and agony of cluelessness, it turned out he just wanted me to test it on other 'toy problems'.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you very much for your answer. I have my TP,FP,FN, TN, ROC area under curve, F-score, how many were classified as male and female, and a confusion matrix. I am looking at doing the McNemar test but not sure where to begin. The formula itself seems straight forward enough though. On wikipedia there is a 2x2 chart example. Could I use this but imagine the chart as the list of my TP,FN,FP,TN? If I just evaluated thousands of instances I'm not exactly sure where to get a,b,c,and d from? I took stats awhile ago but never applied it to something I made myself! $\endgroup$
    – Tastybrownies
    Commented Oct 18, 2013 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.