# Why use maximum likelihood for word prediction?

According to https://www.tensorflow.org/tutorials/word2vec/, the standard approach for predicting the next word in a word sequence is maximum likelihood. The predicted next word is the word that maximizes

$$P(w_t|h)=\frac{\exp(\text{score}(w_t,h))}{\sum_{w'}\exp(\text{score}(w',h))}.$$

To me, it seems that the word predicted from this model is always the word with the maximum score, in which case we could just use the score function for prediction.

What does maximum likelihood provide that a simple score function does not?

That approach is using a score function (in a particular way). It is also using maximum likelihood. Both ways of describing the approach are valid.

If we just want to think about maximizing a score function, the question that raises is -- what score function should we use? How do we choose one? It sounds arbitrary.

In contrast, the maximum-likelihood approach gives us a principled way to choose the function. Here is how to derive that formula, given the maximum-likelihood principle. First, they have apparently decided to use the following approximation for the probability of seeing $w_t$ as the next word, given history $h$:

$$P(w_t,h) = \exp(\text{score}(w_t,h)).$$

Once you've decided to do that, then by the definition of conditional probability it follows that

\begin{align*} P(w_t|h) &= \frac{P(w_t,h)}{P(h)}\\ &= \frac{P(w_t,h)}{\sum_{w'} P(w',h)}\\ &= \frac{\exp(\text{score}(w_t,h))}{\sum_{w'}\exp(\text{score}(w',h))}. \end{align*}

• So in other words, the score function should be one that makes $P(w_t,h)=exp(score(wt,h))$ a valid approximation? Jan 2 '17 at 16:55
• @npCompleteNoob, yes, that's right.
– D.W.
Jan 2 '17 at 18:56