Consensus algorithm for imperfect annotators

Question

I'll pose the question in the abstract first and then describe concretely what I'm trying to do.

Suppose I have a set of 1000 000 multiple-choice questions and 10 oracles that can answer the questions. But the oracles are imperfect and don't always answer correctly. Some oracles answer more accurately than others. We don't know which ones are better beforehand though. All answer correctly with probability greater than 60% and less than 100%.

Suppose each oracle answers 100 questions. One in every 10 questions, however, is not a new question but a (random) question that a different oracle has already answered. Which lets you check whether they each got the same answer or not.

Now my question is this, suppose I want to rank the oracles at the end in order of most reliable to least reliable, what's the best metric? Naively, we could just ask, which oracle had the highest proportion of co-coinciding answers out of all the questions they answered that were also answered by another oracle. But this ignores that fact that if an oracle's answer disagrees with a different oracle, it could be the different oracle that was wrong. So I think there needs to be a way to scale the penalty based on the rating of the oracle that disagreed.

Is there an algorithm for this sort of ranking? The two that came to my mind are the Buchholz system, and page rank. Because they both consider secondary comparisons, not just direct comparisons. But I'm not sure how I could apply them.

Concrete context: I have lots of annotators annotating data. We get to send some of the data to multiple annotators, but not all of it. I'm looking for a way to identify annotators who perform poorly so that I can either remove their annotations or reduce the weight of their annotations.

Answer 1

Technically speaking, you can't. If you flip all of the oracle outputs, then it maintains the same level of agreement but reverses the ranking.

But this is probably a trivial case that's unimportant in practice. Suppose we can assume that every oracle has accuracy that's at least 50%. Then one approach would be to take all questions that are rated by at least three oracles, take the majority vote among all oracles that rated it, and then measure agreement with that majority. I think that should give a correct ordering of the oracles, given enough data (i.e., in the limit, as the number of data points goes to infinity, it should converge to the correct ordering).

In practice you could consider using a more sophisticated model that captures both differences in the accuracy of each oracle and differences in the difficulty of each question. I'm not sure whether this will lead to a major difference in outcomes significantly, but there is tons of work in the research literature you could read about to see what methods have been studied. Here are some example papers:

• Online crowdsourcing: rating annotators and obtaining cost-effective labels, Welinder et al.

• Lean crowdsourcing: Combining humans and machines in an online system, Branson et al.

• Get another label? improving data quality and data mining using multiple, noisy labelers, Sheng et al.

• Max-Margin Majority Voting for Learning from Crowds, Tian et al.

• Active visual recognition with expertise estimation in crowdsourcing, Long et al.

and probably many more - I'm not expert on this topic, so I have no idea whether these are the most important or useful papers, so I suggest doing a literature search of your own.