0
$\begingroup$

I am building a classifier for short texts in a chat system. My features are words and pairs of words.

Naturally, the sentences contain spelling mistakes. If a particular wrong spelling of a certain word hasn't appeared in the training corpus, the classifier has no chance to identify it.

I consider taking an existing spelling corrector, and integrate it with my current classifier, but I am not sure how to do it.

Do you know of a paper that integrates an automatic spelling correction tool with a short text classifier?

|cite|improve this question
$\endgroup$
1
$\begingroup$

Could you not just pre-process the text? Simply pass over the corpus and correct errors via edit-distance or lookup tables. Once this has finished you can then run the classifier.

It you really want to incorporate the correction into your model, it would seem that you can re-define edit distance to compute the log-probability of some misspelled word matching another for some mismatch probability $\theta \sim \text{Beta}(\alpha, \beta)$.

The resulting probabilities can then be incorporated into the classifier directly, which would be parameterized by $\theta$. Depending on how complex your internal model is, you could find the ML or MAP estimate $\hat{\theta}$ by some EM algorithm, or if the model allows, analytically.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ This is an obvious simplification, as the probability of substituting a letter may be different than inserting or leaving it out, but should still give a decent idea on how to incorporate it. $\endgroup$ – Nicholas Mancuso Aug 22 '13 at 15:33
  • $\begingroup$ The problem with pre-processing is that, no speller is perfect. The chatters might use a word that the speller doesn't knowm, and the speller will mistakenly change it. $\endgroup$ – Erel Segal-Halevi Aug 23 '13 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.