# Bloom Filter for 208 million URLs

I need to create a bloom filter of 208 million URLs. What would be a good choice of bit vector size and number of hash functions? I tried a bit vector of size 1 GB and 4 hash functions, but it resulted in too many false positives while reading.

I have a huge web corpus containing web content of billions of URLs. I need to process the web content of URLs satisfying certain criteria: the URL should have appeared in web search results in the past 7 days at least 5 times. This is represented by a list of 208 million URLs. Joining the list directly with the web corpus is not feasible because of volume. So I am considering creation of a bloom filter out of the list and then using the bloom filter to prune out unnecessary URLs from the web corpus.

• Hard to say as you haven't told us your goals etc, making this open ended. You best bet is to experiment. – Aryabhata Sep 19 '12 at 17:58
• @AryaBhata I have a huge web corpus containing web content of billions of URLs. I need to process the web content of URLs satisfying certain criteria (and this is represented by a list of 208 million URLs). Joining the list directly with the web corpus is not feasible because of volume. So I am considering creation of a bloom filter out of the list and then using the bloom filter to prune out unnecessary URLs from the web corpus. – Aadith Ramia Sep 19 '12 at 19:05
• Can you go deeper into the specifics? Like what is actually the criterion? – Aryabhata Sep 20 '12 at 3:10
• Criterion - the URL should have appeared in web search results in the past 7 days at least 5 times – Aadith Ramia Sep 20 '12 at 9:54
• Why don't you edit the question with whatever details you can provide? Perhaps give an example of how you intend to use the bloom filter. – Aryabhata Sep 20 '12 at 15:21

$(1 - (1 - [1/m])^{kn})^k$
where $m$ is the number of bits in the filter, $k$ is the number of hash functions and $n$ is the number of entries in the filter.