I have some difficulties in understanding the intuition behind Naive Bayes Classification.

In general, I do understand every argument of the definition, however I don't understand why it's correct and why any other argument is incorrect.

Start from the beginning:

Goal: $v_{MAP} = argmax_{v_i} p(v_i|a_1...a_n)$

the goal is simply to find the maximum aposteriori of the value of the target function given the set of features Note, here we don't mention the hypothesis, because we are not interested in hypothesis.

After developing the formula with Bayes formula, we get

$argmax_{v_i} p(v_i)p(a_1...a_n|v_i)$

Now, the trick goes $p(a_1...a_n|v_i)=\prod_{j} p(a_j|v_i)$

The question is why in the very begging, having the formula of

$v_{MAP} = argmax_{v_i} p(v_i|a_1...a_n)$

I cannot apply the trick $p(v_i|a_1...a_n)=\prod_{j} p(v_i|a_j)$

It's just a reverse direction, in addition we don't have to calculate the prior probability of $p(v_i)$, so for me it looks like the better way to go.

What's wrong with my reasoning.

  • $\begingroup$ Please proof-read your question. There are numerous errors or typos. Also, I'm not sure what you mean by "every step of the definition" (normally definitions don't have multiple steps), by "why this only chain of steps" (there seems to be a grammar error here). $\endgroup$
    – D.W.
    Dec 22 '13 at 4:34

Because in general $p(v_i \mid a_1 \dots a_n) \ne \prod_j p(v_i | a_j)$. There is no reason to expect that equation to hold with equality. You can't just write down an equation and hope that it is true; you have to be able to justify it from the axioms of probability, and you can't justify the equation you want to hold.

Remember that $p(A \mid B)$ is not the same as $p(B \mid A)$. The two probabilities can be very different. Be careful not to confuse them, or you will definitely get very mixed up.

In general, if $A,B,C$ are events, then $p(A \mid B,C)$ is not necessarily equal to $p(A \mid B) \times p(A \mid C)$. It is easy to come up with a counter-example. For instance, suppose I toss a fair die, and define the following events: $A$ = "the die came up 5", $B$ = "the die is showing an odd number", $C$ = "the die is showing a prime number". Then $p(A \mid B,C) = 1/2$, $p(A \mid B) = 1/3$, $p(A \mid C) = 1/3$, and $1/2 \ne 1/3 \times 1/3$.

The reason why you can write $p(a_1 \dots a_n \mid v_i) = \prod_j p(a_j \mid v_i)$ is because $a_1,a_2,\dots,a_n$ conditionally independent on $v_i$: once $v_i$ is fixed, then $a_1,a_2,\dots,a_n$ become independent. In general, if $A,B$ are conditionally independent on $C$, then $p(A,B \mid C) = p(A \mid C) \times p(B \mid C)$ -- this is in fact the definition of conditional independence. That's why $p(a_1 \dots a_n \mid v_i) = \prod_j p(a_j \mid v_i)$ is justified. (However, no such justification is available for $p(v_i \mid a_1 \dots a_n) \ne \prod_j p(v_i | a_j)$.)

If you are not familiar with the concepts of independence and conditional independence, I suggest reviewing a standard probability textbook, where you can find more on that subject to help you practice and become proficient with those concepts.

  • $\begingroup$ Thank you very much for your answer. I don't understand one point, if I cannot assume that $p(v_i|a_1...a_n) = \prod_{j}^{} p(v_i|a_j)$ why I can assume in Naive Bayes that $p(a_1...a_n|v_j)=\prod_{i}^{}p(a_i|v_j)$. Both of them are assumptions, what the difference between them, why about the second one I can say "simplification" and about the first one I say "incorrect assumption" $\endgroup$
    – user16168
    Dec 22 '13 at 8:11
  • $\begingroup$ @user16168, I've edited my answer to add an explanation of that point. See the edited answer (particularly the last 2 paragraphs). $\endgroup$
    – D.W.
    Dec 22 '13 at 19:48
  • $\begingroup$ finally I understand what do you mean, it's just simple mathematical reasoning, you are aright. I had more philosophical question why I cannot express $p(v_i|a_1...a_n)$ as any combination of $p(v_i|a_j) \forall j$, of course it's not right to use a product as a combination. $\endgroup$
    – user16168
    Dec 23 '13 at 21:25
  • $\begingroup$ @user16168: the reason why you cannot express $p(v|a_1\dots a_n)$ as some function of the $p(v_i|a_j)$ is because, in general, the values of $p(v_i|a_1), \dots, p(v_i|a_n)$ are not sufficient to uniquely determine the value of $p(v|a_1\dots a_n)$. The best we can do is to say $$p(v_i|a_1\dots a_n) = (p(v_i) \prod_j p(a_j|v_i))/(\sum_k p(v_k) \prod_j p(a_j|v_k)) \\= (p(v_i)^{1-n} \prod_j p(a_j) p(v_i|a_j))/(\sum_k p(v_k)^{1-n} \prod_j p(a_j) p(v_k|a_j)).$$ See how the $p(v_i|a_j)$'s aren't enough? You need to know the $p(a_j)$'s and $p(v_k)$'s, too, to compute the latter expression. $\endgroup$
    – D.W.
    Dec 23 '13 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.