0
$\begingroup$

I have a Rails application using Postgresql.

Texts get added to the application (ranging in size -- as short as a few words, to as long as, say, 5,000 words?).

The text gets parsed, both automatically and by hand, to associate each word/position in the text with specific information (part of speech, grammar tags, lemma, definition_id).

Given a lemma, or a part of speech, or grammar tags, or a definition_id (or a combination), I need to be able to find all other text positions in the database that contain the same information (lemma / part of speech / definition_id / etc.).

There are only two options (that I can think of):

  1. A table to store each word position, with columns for each of the above attributes. Finding what I need would be a breeze, my life would be super easy.
  2. A JSON column on the Text object, to store all the word positions. Would be fewer entries, but, a) Constructing a query would be difficult, b) That search would probably be slow, and c) It might take up more storage space than a separate table.

Can I choose option 1? Is storing 1 entry for every word a very bad idea? That’s a lot of entries!

If there is another option I haven’t considered, please enlighten me.

Thanks, Michael King

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ This looks a programming question rather that one on-topic at COMPUTER**SCIENCE**@SE. $\endgroup$ – greybeard May 16 '20 at 10:07
  • $\begingroup$ “...knowledge representation, natural language processing” — guidelines. My question is about structuring a dictionary (for language learning), based off of a natural language processing pipeline (POS tagger). This seems within the bounds of “knowledge representation, and natural language processing”. I have no programming questions, only questions about how to structure this data so that it is an efficient system. $\endgroup$ – michaelsking1993 May 16 '20 at 12:09
  • $\begingroup$ I actually got sent here from StackOverflow 😭 If there is a way that I can simplify my question, or if you’d like to refer me to another caretaker, please let me know. $\endgroup$ – michaelsking1993 May 16 '20 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.