# ML Algorithms to discriminate "known" from "unknown" instances

I'm classifying different electrical devices based on features extracted from their electricity consumption into a household-specific model built from around 10 devices. Now I want to detect whether a classified testing instance is coming from a device which has not been present in the training set, i.e. I want to discriminate known from unknown devices.

I'm working with Weka and use the Classifier.distributionForInstance() method to give me the (classifier-specific) probability distribution for those 10 possible classes for each instance.

Now, the problem is that some classifiers (SMO, NaiveBayes, J48 and other trees) most of the time classify a not known device with a high probability of almost 1 to a class, whereas RandomForest (preferably with high amount of trees) and IBk (kNN) perform rather well in giving back lower probabilities of classification/more uncertain results.

The questions I have: Can you think of alternative approaches or any other algorithms/methods? Why is it, that the first stated algorithms strongly decide for one class. I would have expected them to return unclearer results.

thx, regards

• and btw, should I go to Theoretical Computer Science stackesxchange with that maybe? didn't quite understand what criteria is for that.
– t-h-
Commented Feb 27, 2014 at 13:50
• possibly Cross Validated is better for ML. generally, its important to normalize/condition inputs & look at which inputs have the most info.
– vzn
Commented Feb 27, 2014 at 16:19

As for why Naive Bayes is being overconfident, I would guess this is due to some combination of one of the class priors $P(y)$ being larger than the others, and/or having most features be unknown for your unknown example (so $P(x_i|y)$ is approximately the same for all $y$ and most $i$ if you're smoothing the estimates) and a couple show up in one class causing the $\prod_i P(x_i|y)$ term to blow up for one $y$ relative to the others. This is just a guess without the data.