13
$\begingroup$

A Naive Bayes predictor makes its predictions using this formula:

$$P(Y=y|X=x) = \alpha P(Y=y)\prod_i P(X_i=x_i|Y=y)$$

where $\alpha$ is a normalizing factor. This requires estimating the parameters $P(X_i=x_i|Y=y)$ from the data. If we do this with $k$-smoothing, then we get the estimate

$$\hat{P}(X_i=x_i|Y=y) = \frac{\#\{X_i=x_i,Y=y\} + k}{\#\{Y=y\}+n_ik}$$

where there are $n_i$ possible values for $X_i$. I'm fine with this. However, for the prior, we have

$$\hat{P}(Y=y) = \frac{\#\{Y=y\}}{N}$$

where there are $N$ examples in the data set. Why don't we also smooth the prior? Or rather, do we smooth the prior? If so, what smoothing parameter do we choose? It seems slightly silly to also choose $k$, since we're doing a different calculation. Is there a consensus? Or does it not matter too much?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

The typical reason for smoothing in the first place is to handle cases where $\#\{X_i = x_i | Y = y\} = 0$. If this wasn't done, we would always get $P(Y=y|X=x) = 0$ whenever this was the case.

This happens when, for example, classifying text documents you encounter a word that wasn't in your training data, or just didn't appear in some particular class.

On the other hand, in the case of the class prior probability, $P(Y = y)$, this situation should not occur. If it did this would mean you are trying to assign objects to classes which didn't even appear in the training data.

Also, I've never encountered the term $k$-smoothing. Laplace or Additive smoothing is much more common.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ The reason for smoothing in general is to avoid overfitting the data. The case where the count of some class is zero is just a particular case of overfit (that happens to be particularly bad). You still might want to smooth the probabilities when every class is observed. I suppose I'm bothered by the apparent asymmetry - Laplace smoothing corresponds to assuming that there are extra observations in your data set. Why would you ignore those observations when fitting the prior? $\endgroup$
    – Chris Taylor
    Commented Aug 2, 2012 at 21:09
  • $\begingroup$ I might argue it make less sense to smooth the class prior since the MLE for $P(Y = y)$ is likely to be a lot better than the estimate of $P(X_i = x_i | Y = y)$. If I have reason to believe my class estimates are biased, I'll set aside a validation set and tweak the class priors myself. In my experience, overfitting tends to be a less of a problem with naive Bayes (as opposed to its discriminative counterpart, logistic regression). Perhaps you would prefer or more Bayesian treatment? $\endgroup$
    – alto
    Commented Aug 3, 2012 at 0:04
  • $\begingroup$ "this situation should not occur. If it did this would mean you are trying to assign objects to classes which didn't even appear in the training data". Uhh... how would a classifier assign an object to a class which it had never seen before (ie, isn't in the training data)? $\endgroup$
    – Jemenake
    Commented Apr 29, 2014 at 17:58
  • $\begingroup$ @Jemenake The problem is normally referred to as Zero-shot learning, for example see Zero-Shot Learning with Semantic Output Codes $\endgroup$
    – alto
    Commented Apr 29, 2014 at 18:16
  • $\begingroup$ when we train the model using the training data set, we could build a vocab using the words occur in the training data set, so why not just remove new words not in vocab when make predictions on test set? $\endgroup$
    – avocado
    Commented Sep 28, 2016 at 7:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.