You can take a look at [McNemar's Test](http://en.wikipedia.org/wiki/McNemar%27s_test) or a [Bayesian Testing](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.11.293&rep=rep1&type=pdf). If you are comparing two or more classifiers, you can perform a to evaluate their performances.

Evaluating learning is actually a pretty wide research topic. A good book to refer to would be [Japkowicz-Shah](http://books.google.co.in/books?id=VoWIIOKVzR4C&printsec=frontcover#v=onepage&q&f=false). This is a really nice book on evaluating classifiers and performances. To get some idea about the topic, take a look at [this ICMLA talk](http://www.icmla-conference.org/icmla11/PE_Tutorial.pdf).


If this is a one time thing, get done with McNemar or Bayesian test, however if you continue to work in the field, do see the book.

P.S. ask you boss if he means any real stat tests (like those mentioned above). Once I was working on a classifier, my professor asked me to evaluate its performance, after a long period of pain and agony of cluelessness, it turned out he just wanted me to test it on other 'toy problems'.