2
$\begingroup$

I understand that the output for Latent Dirichlet Allocation is a distribution over K topics.

Suppose I have a Dx(K+1) matrix, where rows are documents and columns are the topic distribution + one column for class. For example, each row represents one movie review. The first K columns represent the topic distribution of the document, and the last column is the classification of this document. For example, if K=5, one row may read as:

2 | 0.25 | 0.4 | 0.1 | 0.15 | 0.1 | 1

where moview review 2 had 25% of the text about topic 1 (pleasure), 40% topic 2 (discontent), 10% topic 3 (personal feelings) ... and the classification of this document was to class #1.

How would I go about creating a Naive Bayes Classifier using this data?

Typically, I have used a Gaussian Naive Bayes where the feature space is iid over normally distributed variables, but this assumption I do not believe makes sense for LDA output. Would I need to assume the features are the individual columns distributed in a particular way (say a Dirichlet probability)?

This exercise is more for proof of concept to use Naive Bayes. I want to use, say 60%, of the data to construct a Naive Bayes Classifier and test the accuracy of the remaining 40%. My main concern is to how to define the PDF for the Naive Bayes Classifier.

$\endgroup$
  • $\begingroup$ @D.W. A NBC to classify each document to a particular class. The classes for example are positive/negative sentiment and the documents are movie reviews. $\endgroup$ – Paul S. Nov 4 '15 at 16:57
  • $\begingroup$ @.D.W. I understand the need to specify this information. I updated the question with the remarks to hopefully help answer the question. $\endgroup$ – Paul S. Nov 4 '15 at 17:08
0
$\begingroup$

The topic scores (the numbers in the first $K$ columns) come from a Dirichlet distribution. The marginals of the Dirichlet distribution are a beta distribution.

Therefore, it would be reasonable to model the distribution on each feature as beta-distributed, not normally-distributed. With this adjustment, all the math for Naive Bayes goes through.


Let me flesh this out. Given some data $x=(x_1,\dots,x_k)$, the Naive Bayes classifier computes the likelihood of a class $c$ as

$$L(c) = P(c) \prod_{i=1}^k P(x_i | c).$$

Here $x_i$ is the value in the $i$th column (the estimated probability the document came from topic $i$). Given the comments above, we will estimate that the conditional distribution $P(x_i | c)$ has a beta distribution with some parameters, i.e., $\text{Beta}(\alpha_i,\beta_i)$. Thus,

$$P(x_i | c) = {1 \over B(\alpha_i,\beta_i)} x_i^{\alpha_i-1} (1-x_i)^{\beta_i-1}$$

for some parameters $\alpha_i,\beta_i$.

Where do we get the parameters from? From the training set, as usual. In other words, we take from the training set just the documents that are classified with class $c=0$, and from those, we fit a beta distribution to the conditional distribution $P(x_i | c=0)$ and find the parameters $\alpha_i,\beta_i$ that make the beta distribution best fit the observed values of $x_i$ (out of the training documents classified as class $c=0$). We use those as our estimate for $P(x_i | c=0)$. How do we find $\alpha_i,\beta_i$? By using standard methods for estimating the parameters of a beta distribution, given a bunch of observations drawn from it.

Then, we do the same for the documents in the training set that are classified as $c=1$, to fit a beta distribution and use that as the distribution for $P(x_i | c=1)$.

Finally, once you have formulas for $P(x_i | c=0)$ and $P(x_i | c=1)$, you plug that into the definition of $L(c)$ above and use that in the Naive Bayes classifier just as normal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.