According to https://www.tensorflow.org/tutorials/word2vec/, the standard approach for predicting the next word in a word sequence is maximum likelihood. The predicted next word is the word that maximizes


To me, it seems that the word predicted from this model is always the word with the maximum score, in which case we could just use the score function for prediction.

What does maximum likelihood provide that a simple score function does not?


That approach is using a score function (in a particular way). It is also using maximum likelihood. Both ways of describing the approach are valid.

Why is it useful to think about this in terms of maximum likelihood? Because it gives a more systematic way to think about this problem and others like it.

If we just want to think about maximizing a score function, the question that raises is -- what score function should we use? How do we choose one? It sounds arbitrary.

In contrast, the maximum-likelihood approach gives us a principled way to choose the function. Here is how to derive that formula, given the maximum-likelihood principle. First, they have apparently decided to use the following approximation for the probability of seeing $w_t$ as the next word, given history $h$:

$$P(w_t,h) = \exp(\text{score}(w_t,h)).$$

Once you've decided to do that, then by the definition of conditional probability it follows that

$$\begin{align*} P(w_t|h) &= \frac{P(w_t,h)}{P(h)}\\ &= \frac{P(w_t,h)}{\sum_{w'} P(w',h)}\\ &= \frac{\exp(\text{score}(w_t,h))}{\sum_{w'}\exp(\text{score}(w',h))}. \end{align*}$$

| cite | improve this answer | |
  • $\begingroup$ So in other words, the score function should be one that makes $P(w_t,h)=exp(score(wt,h))$ a valid approximation? $\endgroup$ – npCompleteNoob Jan 2 '17 at 16:55
  • $\begingroup$ @npCompleteNoob, yes, that's right. $\endgroup$ – D.W. Jan 2 '17 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.