Assume that we have these bigram and unigram data:( Note: not a real data)

#a(start with a) =21
bc= 42
cf= 32
de= 64
e#= 23


# 43

a= 84






what is the probability of generating a word like "abcfde"? I think for having a word starts with a the probability is 21/43. How about bc? is it like bc/b?

  • 1
    $\begingroup$ Have you tried anything yet? What are your thoughts? What calculation approach have you considered? Do you have any specific doubts? Are you familiar with the definition of conditional probability? Have you tried applying that to this problem? Express what you know as a set of probabilities. We want you to do a significant amount of research and self-study before asking, and to show us in the question what you've tried and what specifically you are confused or uncertain about; that helps us help you more effectively. $\endgroup$ – D.W. Apr 11 '16 at 17:40
  • $\begingroup$ I think I answered all of your questions. I read about this subject what I need is just some clarifications. The Hammon book chapter 9 is all about conditional probabilities but there is a question like what I wrote and asked to calculate probabilities with standard maximum likelihood. (which is not made me confuse). $\endgroup$ – liza Apr 11 '16 at 18:03
  • $\begingroup$ I don't understand the problem setup. What is the algorithm you want to feed this input into? $\endgroup$ – Raphael Apr 11 '16 at 21:03

Augment the string "abcde" with # as start and end markers to get #abcde#. Now, as @Yuval Filmus pointed out, we need to make some assumption about the kind of model that generates this data. Because we have both unigram and bigram counts, we can assume a bigram model. In a bigram (character) model, we find the probability of a word by multiplying conditional probabilities of successive pairs of characters, so:

$\Pr[\#abcde\#] = \Pr(a|\#)*\Pr(b|a)*\Pr(c|b)*\Pr(d|c)*\Pr(e|d)*\Pr(\#|e) $

To find the conditional probability of a character $c_2$ given its preceding character $c_1$, $\Pr(c_2|c_1)$, we divide the number of occurrences of the bigram $c_1c_2$ by the number of occurrences of the unigram $c_1$.

So, for example $\Pr(e|d) = count(de)/count(d) = 64/150$

| cite | improve this answer | |
  • $\begingroup$ Yes, that's what I thought. Thank you. $\endgroup$ – liza Apr 13 '16 at 16:40

The probability depends on your generative model for words. Usually it is assumed that there is a probability distribution $p_1$ over the alphabet and another one $p_2$ over pairs of symbols, and the probability of a word is $$ \Pr[w_1\ldots w_n] = p_1(w_1) \prod_{i=2}^n p_2(w_i|w_{i-1}) = p_1(w_1) \prod_{i=2}^n \frac{p_2(w_{i-1}, w_i)}{\sum_\sigma p_2(w_{i-1},\sigma)}. $$ You estimate the unknown probabilities $p_1,p_2$ using unigram and bigram counts. The obvious way would be to estimate $p_1(\sigma) = N_\sigma/N$, where $N$ is the number of occurrences of $\sigma$, and $N = \sum_\sigma N_\sigma$. However, this assigns probability zero for symbols not occurring, so usually $p_1$ is slightly mixed with a uniform distribution. The distribution $p_2$ is estimated in a similar way.

Assuming this generative model, and that there is no mixing, the probability of your word is estimated to be $0$, since (for example) $ab$ never appears in the training data.

| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.