2
$\begingroup$

one of my training questions for my exam is the following one:

Suppose you are testing a new algorithm on a data set consisting of 100 positive and 100 negative examples. You plan to use leave-one-out cross-validation (i.e. 200-fold cross-validation) and compare your algorithm to a baseline function, a simple majority classifier. Given a set of training data, the majority classifier always outputs the class that is in the majority in the training set, regardless of the input. You expect the majority classifier to achieve about 50% classification accuracy, but to your surprise, it scores zero every time. Why?

My only solution about it is that the training data is inverse to the real data. But I'm not sure about my answer. May anybody help me?

Regards,

Patrick

$\endgroup$
3
  • 5
    $\begingroup$ This is a fairly basic question. Do you know what "leave-one-out cross-validation is"? $\endgroup$ Jul 26, 2016 at 14:53
  • $\begingroup$ Yes I do but I thought it's irrelevant because the accuracy is about the majority classifier, isn't it? $\endgroup$ Jul 27, 2016 at 10:40
  • 1
    $\begingroup$ Welcome to Computer Science! The title you have chosen is not well suited to representing your question. Please take some time to improve it; we have collected some advice here. Thank you! $\endgroup$
    – Raphael
    Jul 28, 2016 at 7:24

2 Answers 2

1
$\begingroup$

You have 100 examples of the positive class and 100 examples of the negative class.

Now you do:

examples = "List of all 200 examples"
accuracies = Empty list
for(i=0; i<|examples|; i++) {
    one = examples[i]
    training = examples \ {one}

    # !!!!!!!!!!!!!!!!!!!!!!!!!
    majority_clf = get majority in training. This is the other class than the class of "one"
    # !!!!!!!!!!!!!!!!!!!!!!

    accuracies.append(majority_clf.predict(one) == class(one))
}
overall_accuracy = sum(accuracies) / |accuracies|
$\endgroup$
2
  • 3
    $\begingroup$ What does this code do, and why does it answer the question? Code-only answers are rarely appropriate on this site. $\endgroup$
    – Raphael
    Jul 28, 2016 at 7:25
  • 1
    $\begingroup$ @Raphael Obviously the pseudo code answers the question, as the OP accepted my answer. $\endgroup$ Jul 28, 2016 at 7:47
0
$\begingroup$

Let me try to answer this question with a dummy example that might help you build the intuition. Lets say we have a set of data in the form (xi, yi) with just 10 rows. 5 rows are labeled as 1 and the other ones as 0. Just like in the exercise.

Now, on LOOCV we must extract the first (x1, y1) as the test set and leave the rest as the training set. Assuming that y1 = 1, this means that in this iteration, the training set will have more classes if type yi = 0 than yi = 1, because we took one yi = 1 and we placed it as the test set.

for this iteration, the Majority classifier will be a zero (0) while the test set will be a one (1). They will always be different!

If we run all the "n" iterations, the LOOCV we will have basically the same problem where the test set != majority classifier, in every case, causing the final performance score to be zero (between the LOOCV and the majority classifier). because they will never match.

Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.