I'll pose the question in the abstract first and then describe concretely what I'm trying to do.
Suppose I have a set of 1000 000 multiple-choice questions and 10 oracles that can answer the questions. But the oracles are imperfect and don't always answer correctly. Some oracles answer more accurately than others. We don't know which ones are better beforehand though. All answer correctly with probability greater than 60% and less than 100%.
Suppose each oracle answers 100 questions. One in every 10 questions, however, is not a new question but a (random) question that a different oracle has already answered. Which lets you check whether they each got the same answer or not.
Now my question is this, suppose I want to rank the oracles at the end in order of most reliable to least reliable, what's the best metric? Naively, we could just ask, which oracle had the highest proportion of co-coinciding answers out of all the questions they answered that were also answered by another oracle. But this ignores that fact that if an oracle's answer disagrees with a different oracle, it could be the different oracle that was wrong. So I think there needs to be a way to scale the penalty based on the rating of the oracle that disagreed.
Is there an algorithm for this sort of ranking? The two that came to my mind are the Buchholz system, and page rank. Because they both consider secondary comparisons, not just direct comparisons. But I'm not sure how I could apply them.
Concrete context: I have lots of annotators annotating data. We get to send some of the data to multiple annotators, but not all of it. I'm looking for a way to identify annotators who perform poorly so that I can either remove their annotations or reduce the weight of their annotations.