Best Data Structure For Text Processing

Question

Given a sentence like this, and i have the data structure (dictionary of lists):

{'cat': ['feline', 'kitten']}

A feline was recently spotted in the neighborhood protecting her little kitten

How would I efficiently process these set of text to convert the word synonyms of the word cat to the word CAT such that the output is like this:

A cat was recently spotted in the neighborhood protecting her little cat

I would also like to inquire whether my data structure is relevant and efficient for this task

Answer 1

You can't. At least not always. That's not a well-defined mapping. The notion of synonyms is subjective and a word might have many synonyms it can be mapped into. Even humans can't do that task (they won't always agree what the sentence should be mapped to), so it's not reasonable to expect computers to do it, either.

Perhaps you want to compare two sentences to determine whether they are saying the same thing. In full generality, that is probably AI-complete (i.e., beyond our state of knowledge right now). You can probably get a pretty good approximation by using existing pre-trained encoders that map text (such as a sentence) to an embedding, followed by some distance measure or similarity measure on those embeddings.

Answer 2

Your data structure is quite inappropriate to turn the first sentence into the second. Because you would have to look-up the entire dictionary in search of "feline".

And if, hoping to fix, you would like to replace cat (rather than feline) by a synonym, you don't really know which to choose.

Answer 3

You can do the other way around; use a HashTable to map the synonyms to the main word:

feline -> cat
kitten -> cat

Now, you can do the lookup in $O(1)$ time, although this may have some spatial overhead.