I'm classifying different electrical devices based on features extracted from their electricity consumption into a household-specific model built from around 10 devices. Now I want to detect whether a classified testing instance is coming from a device which has not been present in the training set, i.e. I want to discriminate known from unknown devices.
I'm working with Weka and use the Classifier.distributionForInstance() method to give me the (classifier-specific) probability distribution for those 10 possible classes for each instance.
Now, the problem is that some classifiers (SMO, NaiveBayes, J48 and other trees) most of the time classify a not known device with a high probability of almost 1 to a class, whereas RandomForest (preferably with high amount of trees) and IBk (kNN) perform rather well in giving back lower probabilities of classification/more uncertain results.
The questions I have: Can you think of alternative approaches or any other algorithms/methods? Why is it, that the first stated algorithms strongly decide for one class. I would have expected them to return unclearer results.