4
$\begingroup$

Is there a relatively simple way of telling if two pieces of text are semantically similar?

Some assumptions that are valid:

  • It is all english
  • I have a list of all the important nouns

Are there any strategies that I should pursue? Looking for something that is relatively computationally cheap, though something that could be scaled to improve accuracy at the expense of computational power would be a bonus.

Note:

Assume that there are not enough posts for some type of probabilistic analysis, but some type of NN might be feasible (I think, just don't know enough about it).

Improve this question
$\endgroup$
3
  • $\begingroup$ Wikipedia has some good links and references. $\endgroup$ – Dave Clarke Jul 31 '12 at 11:47
  • $\begingroup$ If you have a small amount of data you'll probably be better off fitting a generative model, rather than a discriminative one, such as most (all?) NN (nearest neighbor) algorithms. That has been my experience and there is some literature to back it up. Have you looked at LSA, pLSA, or LDA? $\endgroup$ – alto Jul 31 '12 at 11:51
  • $\begingroup$ @alto My problem is that there are as many potential categories as there are text samples, and there is no way to know what they will be in the end. How can I deal with this? $\endgroup$ – soandos Aug 3 '12 at 0:42
4
$\begingroup$

Here's a simple technique.

Train an LDA using something like MALLET over your collection of texts. For each pair of documents you want to compare, obtain the topic distributions and compute the Hellinger distance between them.

Things you can tweak include term weighting, the LDA hyperparameters, and the metric for comparing distributions. Term weighting would obviate both the need for a list of important words, and the restriction to only English.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.