Naive Bayes Intuition

Question

I have some difficulties in understanding the intuition behind Naive Bayes Classification.

In general, I do understand every argument of the definition, however I don't understand why it's correct and why any other argument is incorrect.

Start from the beginning:

Goal: $v_{MAP} = argmax_{v_i} p(v_i|a_1...a_n)$

the goal is simply to find the maximum aposteriori of the value of the target function given the set of features Note, here we don't mention the hypothesis, because we are not interested in hypothesis.

After developing the formula with Bayes formula, we get

$argmax_{v_i} p(v_i)p(a_1...a_n|v_i)$

Now, the trick goes $p(a_1...a_n|v_i)=\prod_{j} p(a_j|v_i)$

The question is why in the very begging, having the formula of

$v_{MAP} = argmax_{v_i} p(v_i|a_1...a_n)$

I cannot apply the trick $p(v_i|a_1...a_n)=\prod_{j} p(v_i|a_j)$

It's just a reverse direction, in addition we don't have to calculate the prior probability of $p(v_i)$, so for me it looks like the better way to go.

What's wrong with my reasoning.

Answer 1

Because in general $p(v_i \mid a_1 \dots a_n) \ne \prod_j p(v_i | a_j)$. There is no reason to expect that equation to hold with equality. You can't just write down an equation and hope that it is true; you have to be able to justify it from the axioms of probability, and you can't justify the equation you want to hold.

Remember that $p(A \mid B)$ is not the same as $p(B \mid A)$. The two probabilities can be very different. Be careful not to confuse them, or you will definitely get very mixed up.

In general, if $A,B,C$ are events, then $p(A \mid B,C)$ is not necessarily equal to $p(A \mid B) \times p(A \mid C)$. It is easy to come up with a counter-example. For instance, suppose I toss a fair die, and define the following events: $A$ = "the die came up 5", $B$ = "the die is showing an odd number", $C$ = "the die is showing a prime number". Then $p(A \mid B,C) = 1/2$, $p(A \mid B) = 1/3$, $p(A \mid C) = 1/3$, and $1/2 \ne 1/3 \times 1/3$.

The reason why you can write $p(a_1 \dots a_n \mid v_i) = \prod_j p(a_j \mid v_i)$ is because $a_1,a_2,\dots,a_n$ conditionally independent on $v_i$: once $v_i$ is fixed, then $a_1,a_2,\dots,a_n$ become independent. In general, if $A,B$ are conditionally independent on $C$, then $p(A,B \mid C) = p(A \mid C) \times p(B \mid C)$ -- this is in fact the definition of conditional independence. That's why $p(a_1 \dots a_n \mid v_i) = \prod_j p(a_j \mid v_i)$ is justified. (However, no such justification is available for $p(v_i \mid a_1 \dots a_n) \ne \prod_j p(v_i | a_j)$.)

If you are not familiar with the concepts of independence and conditional independence, I suggest reviewing a standard probability textbook, where you can find more on that subject to help you practice and become proficient with those concepts.