I recently did work in the area of machine learning for one of my jobs and was able to build a classifier which evaluated to an F score of 85%. I also have access to correctly and incorrectly classified instances. My boss would like to make sure my classifier is good and and test it for statistical significance. I'm not quite sure what he means by this. I have taken a course in statistics quite some time ago where I did hypothesis testing and stuff with confidence intervals. Do you think that's what he means or does he want me to compare it with other algorithms? I don't want to say what kind of data set I am working with but it's comparable to something such as predicting spam or ham for email messages.
Any help or guidance on this question would be greatly appreciated. I am pretty new to the area of computer science research. But am I right in saying that statistics are used quite a lot to evaluate these types of things?