3
$\begingroup$

I'm using machine-learning algorithms to solve binary classification problem (i.e. classification can be 'good' or 'bad'). I'm using SVM based algorithms, LibLinear in particular. When I get a classification result, sometimes, the probability estimations over the result are pretty low. For example, I get a classification 'good' with probability of 52% - those kind of results I rather throw away or maybe classify them as 'unknown'.

EDITED - by D.W.'s suggestion

Just to be more clear about it, my output is not only the classification 'good' or 'bad', I also get the confidence level (in %). For example, If I'm the weather guy, I'm reporting that tomorrow it will be raining, and I'm 52% positive at my forecast. In this case, I'm sure you won't take your umbrella when you leave home tomorrow, right? So in those cases where my model does not have a high confidence level I throw away this prediction and don't count it in my estimations.

Unfortunately, I can't find articles regarding thresholding the probability estimations...

Does anyone have an idea what is a normal threshold that I can set over the probability estimations? or at least can refer me to a few articles about it?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

There is no universal answer. Instead, it depends on your application. What counts as useful for your application? That determines what should count as a useful or good-enough machine learning algorithm. What counts as useful will vary widely from application to application; some applications require 99.99% accuracy, others might be happy with 52% accuracy.

In practice, there are also multiple ways to define accuracy. If you expect that (in the ground truth) half of the objects should be classified 'good' and half as 'bad', then the obvious accuracy measure suffices, i.e., counting what fraction of instances your classifier outputs the correct answer. However, if 'good' objects are a lot more common than 'bad' objects, or if the penalty for mis-classifying a 'good' object as 'bad' is very different from the penalty for mis-classifying a 'bad' object as 'good', then you might want to measure accuracy differently. The best measure of accuracy is again application-dependent; there is no single universal answer.

Improve this answer
$\endgroup$
6
  • $\begingroup$ you're right but were you trust a model that tells you that for some test its prediction is true with probability of 52%? maybe I'll try to rephrase: on how many results would you give away (also correct ones) for a better certainty? or confidence? References for that would be also happily accepted $\endgroup$
    – Ziv Levy
    Commented Dec 20, 2013 at 22:49
  • 1
    $\begingroup$ @ZivLevy, the question of whether to trust your estimate of the accuracy of the model is a separate one. The way you test that is by setting aside a separate subset of the data for model validation (don't use it for training or anything else), and then evaluate your model on that separate subset of data. But in general I'm inclined to agree with you: if random guessing gets you 50% accuracy, and your learned model gets 52% accuracy, my first reaction would be to worry about over-fitting and suspect that the model might not be very useful. But the way to test that concern is as I described. $\endgroup$
    – D.W. ♦
    Commented Dec 20, 2013 at 22:56
  • $\begingroup$ @ZivLevy, I confess I don't understand what you mean by "how many results would you give away". Also, when you say better certainty/confidence: certainty of what? Certainty that your model really is 52% accurate? $\endgroup$
    – D.W. ♦
    Commented Dec 20, 2013 at 22:56
  • $\begingroup$ maybe its my fault and I should have mentioned at the post: my output is not only the classification 'good' or 'bad', I also get the confidence level (in %). For example, If I'm the weather guy, I'm reporting that tomorrow it will be raining, and I'm 52% positive at my forecast. In this case, I'm sure you won't take your umbrella when you leave home tomorrow, right? So in those cases where my model does not have a high confidence level I throw away this prediction and don't count it in my estimations. $\endgroup$
    – Ziv Levy
    Commented Dec 20, 2013 at 23:06
  • $\begingroup$ @ZivLevy, OK. Well, I think you need to go back and edit your question. The format of this site might be a bit new to you, so let me elaborate. This is a Q&A site, not a discussion forum, which often trips many new users up. We expect you to ask a carefully-posed, well-research question with all the context and information needed in the body of the question. If it takes an extended comment thread to get to the heart of what you're asking, you didn't pose the question clearly enough. In any case, now is the perfect time for you to go edit it to make it self-contained! $\endgroup$
    – D.W. ♦
    Commented Dec 20, 2013 at 23:30
2
$\begingroup$

If your method is reasonable, then there is a correlation between your confidence and accuracy rates---when your confidence is higher so is your accuracy. Then, get some sample from the true distribution to use as a gold standard. Now study the mentioned correlation. What is the threshold of confidence at which you start to lose accuracy?

Improve this answer
$\endgroup$
0
$\begingroup$

For each class you will have a gain $G$ for getting a prediction correct and a loss $L$ for getting a prediction wrong. You solve for probability p using the equation $Gp - L(1-p) = 0$ to get $p$ for which any confidence below should not be scored i.e. $G = 70, L=10, p = 12.5\%$. Each class will have a different threshold.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I like what you're trying to do in this answer! However, it seems like we also need to take into account the gain when we decide not to make any prediction at all, and compare the expected gain if we do make a prediction against the expected gain if we don't. Perhaps you're assuming the gain is 0 if we don't make a prediction? Separately: Hopefully, a binary classification algorithm should never have a confidence $p$ below 0.5; if $p<0.5$, swap the prediction (the label output by the classifier) and replace $p$ with $1-p$, and you have a better prediction. $\endgroup$
    – D.W. ♦
    Commented Jun 21, 2017 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.