To frame the question, in computer science often we want to calculate the product of several probabilities:
P(A,B,C) = P(A) * P(B) * P(C)
The simplest approach is simply to multiply these numbers, and that is what I was going to do. However, my boss said it's better to add the log of the probabilities:
log(P(A,B,C)) = log(P(A)) + log(P(B)) + log(P(C))
This gives the log probability, but we can get the probability afterwards if necessary:
P(A,B,C) = e^log(P(A,B,C))
Log addition is considered better for two reasons:
- It prevents "underflow" whereby the product of probabilities is so small that it gets rounded to zero. This can often be a risk since probabilities are often very small.
- It is faster because many computer architectures can perform addition more quickly than multiplication.
My question is about the second point. This is how I have seen it described, but it doesn't take into account the added cost of getting the log! We should be comparing "cost of log + cost of addition" to "cost of multiplication". Is it still smaller after taking that into account?
Also, the Wikipedia page (Log probability) is confusing in this respect, stating "The conversion to log form is expensive, but is only incurred once." I don't understand this, because I think you would need to take the log of every term independently before adding. What am I missing?
Finally, the justification that "computers perform addition faster than multiplication" is kind of vague. Is that specific to the x86 instruction set, or is it some more fundamental trait of processor architectures?