Consider a system that indexes documents based on vector space model and a simple query, such as
qwe asd. When searching we assign weights to both words
asd based on how often they appear in the index. The idea being that if word exists in larger number of documents then it should have smaller impact on relevance. This works well because for each word we usually store statistics: number of occurrences of a given word in the whole index and number of occurrences of a given word in each document (possibly after normalization).
Now consider another query:
"qwe asd" zxcrty. We have two parts here: a phrase and a simple word. For the word we have the above mentioned statistics, but for the phrase we do not. This poses a question: how to rank documents against phrase searches? If we find a one document that contains the phrase (
qwe asd) and another that contains the single word (
zxcrty) which one should be ranked higher?
I somehow doubt that there is one and ultimate solution to this question but would like to know about what approaches are used in existing search engines, whether other models solve the issue and other information that can be useful in analysing the problem.