Given a dictionary of words and their frequencies (how many times they appear in a universe and given a string(no spaces, punctuation, etc.). What is the best way to segment into individual words? I believe a dynamic programming approach would be best. But I am not sure how to proceed about throwing away unnecessary prefixes of identified words and so on.
Ex. "theyouthevent" could be "the you the vent" or "they out he vent" or "the youth event" (most likely)
I think Peter Norvig outlines a pretty good approach in this http://norvig.com/ngrams/ch14.pdf
Just curious to see if there are any other good approaches suited for the specific problem with the data I have. Norvig has n-gram data while I only have word frequencies (which is essentially a unigram)