I have an extremely large (100GB+) corpus of many different texts. All of them are in English and 'well' formatted. They are not loaded into any kind of database, think of them as a huge collection of text files of many different sizes.
What would be the most efficient way to check and retrieve arbitrary text sequences that may be present in the larger corpus? (Precomputation is allowed).
Is there some kind of hash function that I could use and then compare with my database?
I'm looking for the fastest possible implementation.