1
$\begingroup$

I have an extremely large (100GB+) corpus of many different texts. All of them are in English and 'well' formatted. They are not loaded into any kind of database, think of them as a huge collection of text files of many different sizes.

What would be the most efficient way to check and retrieve arbitrary text sequences that may be present in the larger corpus? (Precomputation is allowed).

Is there some kind of hash function that I could use and then compare with my database?

I'm looking for the fastest possible implementation.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Welcome to CS.SE! Can you edit the question to tell us more? When you say "text sequences", do you mean a substring query? Or perhaps you are asking about a sequence of words? How large will the queries be? What approaches have you already considered? How much do you care about the space required to store the precomputed data, or the amount of time/space needed for the precomputation? Have you looked at suffix trees/arrays? $\endgroup$ – D.W.♦ Sep 6 '17 at 4:55
  • $\begingroup$ Postgre has full text search maybe that's sufficient for what you want to do. $\endgroup$ – adrianN Sep 6 '17 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.