In order to select questions for an online contest, we get contributors to submit potential questions that they write themselves. Then, out of, say 100 submitted questions, we have to rate them and select the best ones, say about 30. Suppose we have 4-5 people willing to help on the final selection task. No one has time to go though all 100 proposals, but can give a grade to a subset (say, 30-35).

We can assume those raters are internally consistent (i.e., will always give the same rate to the same question) but not with each other (e.g., there may be a systematic bias between two raters).

What algorithms are there to help assign questions to raters and them try to estimate and adjust the biais so as to obtain a maximally consistent global rating without anyone having to read all the questions?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.