Skip to main content
added 1687 characters in body
Source Link
D.W.
  • 165.6k
  • 21
  • 230
  • 490

The topic scores (the numbers in the first $K$ columns) come from a Dirichlet distribution. The marginals of the Dirichlet distribution are a beta distribution.

Therefore, it would be reasonable to model the distribution on each feature as beta-distributed, not normally-distributed. With this adjustment, all the math for Naive Bayes goes through.


Let me flesh this out. Given some data $x=(x_1,\dots,x_k)$, the Naive Bayes classifier computes the likelihood of a class $c$ as

$$L(c) = P(c) \prod_{i=1}^k P(x_i | c).$$

Here $x_i$ is the value in the $i$th column (the estimated probability the document came from topic $i$). Given the comments above, we will estimate that the conditional distribution $P(x_i | c)$ has a beta distribution with some parameters, i.e., $\text{Beta}(\alpha_i,\beta_i)$. Thus,

$$P(x_i | c) = {1 \over B(\alpha_i,\beta_i)} x_i^{\alpha_i-1} (1-x_i)^{\beta_i-1}$$

for some parameters $\alpha_i,\beta_i$.

Where do we get the parameters from? From the training set, as usual. In other words, we take from the training set just the documents that are classified with class $c=0$, and from those, we fit a beta distribution to the conditional distribution $P(x_i | c=0)$ and find the parameters $\alpha_i,\beta_i$ that make the beta distribution best fit the observed values of $x_i$ (out of the training documents classified as class $c=0$). We use those as our estimate for $P(x_i | c=0)$. How do we find $\alpha_i,\beta_i$? By using standard methods for estimating the parameters of a beta distribution, given a bunch of observations drawn from it.

Then, we do the same for the documents in the training set that are classified as $c=1$, to fit a beta distribution and use that as the distribution for $P(x_i | c=1)$.

Finally, once you have formulas for $P(x_i | c=0)$ and $P(x_i | c=1)$, you plug that into the definition of $L(c)$ above and use that in the Naive Bayes classifier just as normal.

The topic scores (the numbers in the first $K$ columns) come from a Dirichlet distribution. The marginals of the Dirichlet distribution are a beta distribution.

Therefore, it would be reasonable to model the distribution on each feature as beta-distributed, not normally-distributed. With this adjustment, all the math for Naive Bayes goes through.

The topic scores (the numbers in the first $K$ columns) come from a Dirichlet distribution. The marginals of the Dirichlet distribution are a beta distribution.

Therefore, it would be reasonable to model the distribution on each feature as beta-distributed, not normally-distributed. With this adjustment, all the math for Naive Bayes goes through.


Let me flesh this out. Given some data $x=(x_1,\dots,x_k)$, the Naive Bayes classifier computes the likelihood of a class $c$ as

$$L(c) = P(c) \prod_{i=1}^k P(x_i | c).$$

Here $x_i$ is the value in the $i$th column (the estimated probability the document came from topic $i$). Given the comments above, we will estimate that the conditional distribution $P(x_i | c)$ has a beta distribution with some parameters, i.e., $\text{Beta}(\alpha_i,\beta_i)$. Thus,

$$P(x_i | c) = {1 \over B(\alpha_i,\beta_i)} x_i^{\alpha_i-1} (1-x_i)^{\beta_i-1}$$

for some parameters $\alpha_i,\beta_i$.

Where do we get the parameters from? From the training set, as usual. In other words, we take from the training set just the documents that are classified with class $c=0$, and from those, we fit a beta distribution to the conditional distribution $P(x_i | c=0)$ and find the parameters $\alpha_i,\beta_i$ that make the beta distribution best fit the observed values of $x_i$ (out of the training documents classified as class $c=0$). We use those as our estimate for $P(x_i | c=0)$. How do we find $\alpha_i,\beta_i$? By using standard methods for estimating the parameters of a beta distribution, given a bunch of observations drawn from it.

Then, we do the same for the documents in the training set that are classified as $c=1$, to fit a beta distribution and use that as the distribution for $P(x_i | c=1)$.

Finally, once you have formulas for $P(x_i | c=0)$ and $P(x_i | c=1)$, you plug that into the definition of $L(c)$ above and use that in the Naive Bayes classifier just as normal.

Source Link
D.W.
  • 165.6k
  • 21
  • 230
  • 490

The topic scores (the numbers in the first $K$ columns) come from a Dirichlet distribution. The marginals of the Dirichlet distribution are a beta distribution.

Therefore, it would be reasonable to model the distribution on each feature as beta-distributed, not normally-distributed. With this adjustment, all the math for Naive Bayes goes through.