I'm both interested in practical and theoretical aspects of the problem. How do you store linguistic data. Suppose you have stemmed, parsed, tagged and, perhaps even did some preliminary analysis, such as clustering by topics, how then do you store this information for reuse?

I've been looking at knowledge representation formats, but, and maybe because I was doing it on my own, without guidance, two aspects strike me as odd:

  1. The technology is very immature (and most things I could find relevant to the subject seem to be abandoned in the last century).
  2. Even then, there weren't real storage engines. The closest I could get was a triplestore. While, maybe at the basis level this is an interesting idea, there seem to be no tools or techniques which build on top of it.

To make this question more concrete, consider this chain: there are relations out there in the real world, certain aspects of them are captured by relational algebra; SQL language builds on top of relational algebra, adding new concepts such as tables, triggers etc.

Similarly, out there in the world there are phrases people say or write to each other, there are several theories which try to capture the underlying principles of the structure of those phrases (for example, dependency parsing or constituency parsing). If the later is in some sense equivalent to relational algebra, what would be the equivalent of SQL-enabled databases for linguistic data?

  • $\begingroup$ Your question is considering so may aspects of linguistic data processing that it is hard to understand what you are really concerned with, or to give a uniform answer to your question. What kind of information do you want to store, and for what purpose? The first issue to store information is to have a storage format, and that is of course dependent on what you are storing. A second point is that data that is produced mechanically may not need to be stored. Why should it make sense to store parsing structures as SQL database, and how would it relate with semantic information $\endgroup$ – babou Jun 5 '15 at 12:12
  • $\begingroup$ For example, how do you propose to deal with ambiguity? You may of course ignore it, as many systems do, Would that be a form of immaturity? $\endgroup$ – babou Jun 5 '15 at 12:12
  • $\begingroup$ @babou Bottom-up: I want to store data in a database because (1) it's simply caching (2) imposing structure on data will hopefully yield new insights or techniques of processing it (possibly with different tools). To make it more concrete: I'm parsing IRC logs and I want to organize a knowledge base from the parse, such as it could be queried for subjects discussed in the chat. So, do I have a storage format?--No, I have dozens I can choose from, but before I commit to the choice, I want to know that the format has tools to work with it. And, even if no tools, then, at least a theory of tools. $\endgroup$ – wvxvw Jun 5 '15 at 12:37
  • $\begingroup$ I do not think imposing a format on data is the way to get insight about its fundamental properties (barring serendipity, which is always possible). It is for you to select what matters, but as you seem refusing to do that, I wonder whether your question has a proper answer. $\endgroup$ – babou Jun 5 '15 at 12:47
  • $\begingroup$ @babou simple example of what I mean by imposing a format / gaining insight: suppose I collected from each student their name, and from the teacher, I've collected information on how student name relates to test id, and how test id relates to the grade. Using relational database, I can then infer relation between name and grade. The format of relational database made it trivial to make such inference. I'm looking for a framework that at least simplifies the process of making similar inferences about chunks of natural language. $\endgroup$ – wvxvw Jun 5 '15 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.