I'm thinking about the optimal algorithm for the following problem:
- a text, say it's an article about 5-50 pages.
- a set of ngrams (ngram strings, n>2), of arbitrary length, could be more than 20k n-grams.
The algorithm should output the following:
- a dictionary of all ngrams that were found in the text with the corresponding quantities, it should also take into account that ngrams could partially intersect or consist of each other (like 'probability density', 'probability density function', 'probability density distribution')
So the question is what would be the most time-efficient algorithm to compute this?
Both all words in a text and all words in ngrams are reduced to the canonical forms.