0
$\begingroup$

My goal is to recommend jobs to job seekers based on their skill set.

Currently I'm using an SVM for this, which is outputting one prediction, e.g. "software engineer at Microsoft". However, consider this: how significantly different are the skill sets of a software engineer at Microsoft and a software engineer at IBM? Probably not significantly different. Indeed, by inspection of my data set I can confirm this. Hence, the SVM struggles to discriminate in situations like this, of which there are many in my data set, and my classification accuracy is about 50%.

So I had an idea.

In SK Learn, once you've trained some model, you can compute the probability a particular input X belongs to each class.

So for each input X in my test set, I took the the top 3 most likely classifications. Then I tested whether or not the correct label was in the top 3 predictions. If it was, then I considered the prediction to be correct. In doing so, the classification accuracy increased to over 80%.

So my question is: is this a valid approach to measuring classification accuracy? If it is, then does it have a name?

In my mind, it is valid given my intended application, which is to recommend a selection of jobs to a job seeker, which are relevant to their skill set.

Improve this question
$\endgroup$
2
  • $\begingroup$ Cross-posted: datascience.stackexchange.com/q/63875/8560. Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. $\endgroup$
    – D.W. ♦
    Commented Nov 27, 2019 at 20:33
  • 1
    $\begingroup$ This is a valid approach insofar as you accept a non-classification: now instead of one class, you get three to choose from. Hence claiming 80% classification accuracy is a fallacy. (Reporting all classes yields 100% accuracy.) $\endgroup$
    – user16034
    Commented Aug 14, 2022 at 15:15

1 Answer 1

Reset to default
0
$\begingroup$

This is called top-3 accuracy. See https://stats.stackexchange.com/q/95391/2921, https://stackoverflow.com/q/37668902/781723, https://stats.stackexchange.com/q/156471/2921. Sure, you can use it. Whether top-1 accuracy or top-3 accuracy is better correlated with user satisfaction will depend on your particular application and on your users.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.