I am trying to implement a natural language query preprocessing module which would, given a query formulated in natural language, extract the keywords from that query and submit it to an IR (information retrieval system) system.

At first, I thought about using some training set to compute TF-IDF values of terms and use these values for estimating the importance of single words. But on second thought, this does not make any sense in this scenario - I only have a training collection but I dont have access to index the indexed IR data. Would it be reasonable to only use the IDF value for such estimation (is IDF enough to establish weight of a term in general)? Or maybe another weighting approach?

Could you suggest how to tackle this problem? Usually, the articles about NLP processing that I read talks about training and test data sets. But what if I only have the query and training data?

  • 1
    $\begingroup$ I don't really understand your question :( could you please add some more detail. TF-IDF can be calculated on any set of documents. You give me a set of webpages then I can calculate the TF-IDF weighting for all the words in them, period. $\endgroup$
    – jhegedus
    Nov 23, 2014 at 19:58
  • $\begingroup$ Please update with $\text{Input}$ and $\text{Output}$ examples. $\endgroup$ Feb 21, 2015 at 22:34

1 Answer 1


I don't know the relative sizes and the nature of your problem, these details could completely change the view. But generally, if the only thing you have at your disposal is a small set of test documents, I would not recommend using that for term weighting at all. A small set of documents would give you an illusion of coverage, a skewed set of weights that would not accurately cover the domain. If you do have general access to the IR system, though, you could try using the system itself for obtaining statistics by hitting it with random words and recording the number of search results, for example.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.