0
$\begingroup$

I am working through the NLP notes for Naive Bayes' classification here: https://web.stanford.edu/~jurafsky/slp3/6.pdf

Below $c$ is the class of the observation and $w_i$ is the $i$th word of a text document.

In developing the maximum likelihood estimate for the conditional probability of the feature (word) $w_i$ given class $c$ the following expression is given for the conditional probability:

$$\hat{P}(w_i \mid c) = \frac{count(w_i,c)}{\sum_{w\in V} count(w,c)}$$

It is mentioned that $V$ consists of the union of all the word types in all classes, not just the words in one class $c$. The importance of this fact is stressed later on when add-one (Laplace) smoothing is used to avoid probability estimates that equal 0.

It is mentioned that:

It is crucial that the vocabulary $V$ consists of the union of all the word types in all of the classes, not just the words in one class $c$ (try to convince yourself that this is true).

I have thought for awhile on this aside given in the text, but it does not seem to make much sense. To me, wouldn't $count(w,c)$ always equal $0$ for any $w$ which is not contained in a document of class $c$? Why sum over all words in the vocabulary if they will just equal $0$?

$\endgroup$
1
$\begingroup$

If you're using Laplace smoothing, you'll replace the count with a "pseudocount", and that won't be zero. So, it does make a difference.

$\endgroup$

This site is temporarily in read only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .