In NLP, do you distinguish tokens that you don't observe in a training sample and still expect that they may occur in a test sample, between

  • those you know what they are, and
  • those you don't know what they are, and how many of them

If yes, how do you treat them differently to estimate the probabilities in N-grams?

  • $\begingroup$ Have you heard of Laplacian smoothing and Bayesian smoothing? If not, do some research on them... $\endgroup$ – D.W. Feb 27 '14 at 21:12
  • $\begingroup$ yes. I know them. But for those words you don't see in the training sample, and don't know who they are, and how many of them, I don't know how to apply Laplace smoothing and Bayesian smoothing. The methods seem only apply to the case that you know what they are and how many of them. $\endgroup$ – Tim Feb 27 '14 at 21:20
  • $\begingroup$ Perhaps you might want to edit your question to explain what techniques you have already considered and why they don't seem to apply to your situation? $\endgroup$ – D.W. Feb 27 '14 at 21:45
  • $\begingroup$ I don't have techiques to solve the second case. $\endgroup$ – Tim Feb 27 '14 at 22:58
  • $\begingroup$ Again, rather than adding comments here, I recommend that you edit the question. On this site, we expect the question to stand alone; people should not have to read the comments to get everything. Comments exist solely to help you improve the question. We also expect that you'll tell us what you tried, and if you know of some seemingly-relevant approach that doesn't actually solve your problem, we'd normally expect you to mention it in the question and explain why it doesn't actually solve your problem. So, please click the "edit" button underneath your question and improve it. $\endgroup$ – D.W. Feb 28 '14 at 0:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.