Computer Science Stack Exchange is a question and answer site for students, researchers and practitioners of computer science. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am implementing a Naive Bayes algorithm for text categorization with Laplacian smoothing. The problem I am having is that the probability approaches zero because I am multiplying many small fractions. Therefore, the probability eventually yields zero. This is because there are several words within the documents and training sets.

Because of this, I am not able to categorize the texts. Is there a way I can get around this problem? Am I doing something wrong in my implementation?

share|cite|improve this question
    
... You could avoid floating-point arithmetic. ​ ​ – Ricky Demer Mar 16 at 9:51

The usual trick to avoid this underflow is to compute with logarithms, using the identity $$ \log \prod_{i=1}^n p_i = \sum_{i=1}^n \log p_i. $$ That is, instead of using probabilities, you use their logarithms. Instead of multiplying them, you add them.

Another approach, which is not so common, is to normalize the product manually. Instead of keeping just one floating point number $p$, you keep a floating point number $p_0 \in [1,2)$ (say) and a negative exponent $x$ such that $p = p_0 2^x$. After each operation you normalize the resulting number.

share|cite|improve this answer
2  
It is also useful to note the logsumexp trick in this context: en.wikipedia.org/wiki/LogSumExp – Bitwise Mar 14 at 23:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.