How do IR researchers evaluate the ranks of documents?

Question

I am developing a new IR system in a specialized context. I understand that a traditional IR system (like a search engine) should rank documents in terms of their relevance for a query. The most relevant documents should come first and the least relevant (perhaps: least relevant above some threshold) should come last.

I want to evaluate my new IR system. How do researchers evaluate document rank? How do they say that this document is more relevant than that document for some query? The most obvious thing to do would be to manually assign such labels and then check the machine against the hand labels. This seems highly subjective. Maybe there is a better way?

I've read section 15.1 Manning's Foundations of Natural Language Processing but it only talks about evaluating precision and recall -- not evaluating rank. Any suggestions on where to look on evaluating rank?

Answer 1

If I understand what you're trying to achieve correctly, you can use this technique discounted cumulative gain.

$$DCG_p=\sum_{i=1}^{p}{\frac{2^{rel_i}-1}{\log_2{(i+1)}}}$$

$i$ is the rank, and $p$ is the number of results that you want to evaluate. For example, if you evaluate DCG for the first 10 results, then $p=10$.

NDCG is a form of DCG that involves normalizing the result. This is useful if p differs when you compare DCG scores. $$NDCG=\frac{DCG}{IDCG}$$ where IDCG is the ideal DCG. This is found by calculating the DCG up to $p$ under ideal conditions (all documents have a relevance of 1).

Note that neither recall nor precision are used here. Relevance and rank are the factors to compare. You still need to give the documents a relevance factor, that's unavoidable, but you don't need to establish a threshold as you would with other evaluation metrics.