I'm studying for an exam and came across this question. I feel like it's much simpler than I am making it out to be, but I'm not sure.

Suppose we have a corpus in which Zipf’s Law holds. If the most frequent term occurs one million times, and the next most frequent term occurs 250,000 times, how often should we expect to see the third most frequent term? And the fourth most frequent term?

Wikipedia has the formula $$f\left(k,s,N\right)=\frac{\frac{1}{k^{s}}}{\sum\limits_{n=1}^{N}\frac{1}{n^{s}}},$$ We but in this problem, $N$ (the number of terms) and $s$ (the value of the exponent characterizing the distribution) aren't given. Only $k$ (the rank) is given and the frequencies of each rank. I know that Zipf's Law says that the frequency of a term is inversely proportional to the term's rank, but I don't know what that constant factor would be in $f_{t}=c\cdot\frac{1}{k}$.

  • $\begingroup$ The title you have chosen is not well suited to representing your question. Please take some time to improve it; we have collected some advice here. Thank you! $\endgroup$ – Raphael Mar 28 '18 at 10:11

The question gives you enough information to deduce $N$ and $s$. You are given $f(1,s,N)$ and $f(2,s,N)$. Note that $$ \frac{f(1,s,N)}{f(2,s,N)} = \frac{1/1^s}{1/2^s} = 2^s. $$ Hence you can find $s$. In order to calculate $f(3,s,N),f(4,s,N)$ you don't really need to find $N$, since, $$ \frac{f(1,s,N)}{f(3,s,N)} = 3^s, \quad \frac{f(1,s,N)}{f(4,s,N)} = 4^s, $$ using calculations similar to what we had before.

Notice that in order to solve this exercise, we had to know absolutely nothing about the Zipf law other than the formula. Stated differently, what you need in order to solve this exercise is not knowledge (apart from the formula for $f(k,s,N)$) but rather problem solving skills. One of the main goals of the undergraduate degree is to develop such skills.

| cite | improve this answer | |
  • $\begingroup$ Oh wow that was much simpler than I thought. I appreciate your help. Thanks! $\endgroup$ – Jonathan Barkey Mar 28 '18 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.