13
$\begingroup$

A Naive Bayes predictor makes its predictions using this formula:

$$P(Y=y|X=x) = \alpha P(Y=y)\prod_i P(X_i=x_i|Y=y)$$

where $\alpha$ is a normalizing factor. This requires estimating the parameters $P(X_i=x_i|Y=y)$ from the data. If we do this with $k$-smoothing, then we get the estimate

$$\hat{P}(X_i=x_i|Y=y) = \frac{\#\{X_i=x_i,Y=y\} + k}{\#\{Y=y\}+n_ik}$$

where there are $n_i$ possible values for $X_i$. I'm fine with this. However, for the prior, we have

$$\hat{P}(Y=y) = \frac{\#\{Y=y\}}{N}$$

where there are $N$ examples in the data set. Why don't we also smooth the prior? Or rather, do we smooth the prior? If so, what smoothing parameter do we choose? It seems slightly silly to also choose $k$, since we're doing a different calculation. Is there a consensus? Or does it not matter too much?

$\endgroup$
5
$\begingroup$

The typical reason for smoothing in the first place is to handle cases where $\#\{X_i = x_i | Y = y\} = 0$. If this wasn't done, we would always get $P(Y=y|X=x) = 0$ whenever this was the case.

This happens when, for example, classifying text documents you encounter a word that wasn't in your training data, or just didn't appear in some particular class.

On the other hand, in the case of the class prior probability, $P(Y = y)$, this situation should not occur. If it did this would mean you are trying to assign objects to classes which didn't even appear in the training data.

Also, I've never encountered the term $k$-smoothing. Laplace or Additive smoothing is much more common.

$\endgroup$
  • 1
    $\begingroup$ The reason for smoothing in general is to avoid overfitting the data. The case where the count of some class is zero is just a particular case of overfit (that happens to be particularly bad). You still might want to smooth the probabilities when every class is observed. I suppose I'm bothered by the apparent asymmetry - Laplace smoothing corresponds to assuming that there are extra observations in your data set. Why would you ignore those observations when fitting the prior? $\endgroup$ – Chris Taylor Aug 2 '12 at 21:09
  • $\begingroup$ I might argue it make less sense to smooth the class prior since the MLE for $P(Y = y)$ is likely to be a lot better than the estimate of $P(X_i = x_i | Y = y)$. If I have reason to believe my class estimates are biased, I'll set aside a validation set and tweak the class priors myself. In my experience, overfitting tends to be a less of a problem with naive Bayes (as opposed to its discriminative counterpart, logistic regression). Perhaps you would prefer or more Bayesian treatment? $\endgroup$ – alto Aug 3 '12 at 0:04
  • $\begingroup$ "this situation should not occur. If it did this would mean you are trying to assign objects to classes which didn't even appear in the training data". Uhh... how would a classifier assign an object to a class which it had never seen before (ie, isn't in the training data)? $\endgroup$ – Jemenake Apr 29 '14 at 17:58
  • $\begingroup$ @Jemenake The problem is normally referred to as Zero-shot learning, for example see Zero-Shot Learning with Semantic Output Codes $\endgroup$ – alto Apr 29 '14 at 18:16
  • $\begingroup$ when we train the model using the training data set, we could build a vocab using the words occur in the training data set, so why not just remove new words not in vocab when make predictions on test set? $\endgroup$ – avocado Sep 28 '16 at 7:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.