Distinct elements count of huge multiset - Computer Science Stack Exchange most recent 30 from cs.stackexchange.com 2019-08-22T06:59:39Z https://cs.stackexchange.com/feeds/question/106475 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://cs.stackexchange.com/q/106475 3 Distinct elements count of huge multiset Cedric Martin https://cs.stackexchange.com/users/102485 2019-04-04T14:49:24Z 2019-04-10T23:23:53Z <p>I know that <em>HyperLogLog</em> can approximate the distinct elements count of a huge multiset but I was wondering if it was possible, using a method I saw mentioned on an IRC channel, to get an exact answer while still using significantly less space than a traditional approach.</p> <p>Would the following work to compute the exact (not approximate) distinct elements count of a huge multiset?</p> <p>The idea is to use a <em>Bloom filter</em> twice, first processing the data one way, and then in reverse and additionally using a map (whose size is much smaller than the huge multiset -- this property is provided by the Bloom filter).</p> <pre><code>1st pass create an empty bloom filter create an empty map for each element from the multiset if already in the bloom add to the map with key element / value 0 else add to the bloom </code></pre> <p>Second pass is nearly identical but elements from the huge multiset are processed in reverse (multiset doesn't need to be an ordered set: it simply needs to have its elements iterated from the end to its beginning).</p> <pre><code>2nd pass empty the bloom filter (but not the set) for each element from the multiset (but processed this time in reverse order) if already in the bloom add to the map with key element / value 0 else add to the bloom </code></pre> <p>At this point the map contains a list of potentially, but not necessarily, clashing entries in the multiset. So the entries are all set to zero.</p> <pre><code>3rd pass cnt = 0 for each element in the multiset if present in the map, increment value in that map for that key else increment cnt </code></pre> <p>Exact cardinality is now equal to <code>"cnt + nb entries in the map whose value is 1 or greater"</code>. </p> <p>This requires three passes but uses space identical to a <em>Bloom filter</em> plus a map to count the x% of entries the <em>Bloom filter</em> couldn't answer if they were present or not.</p> <p>I hacked a quick proof of concept and did some generative testing and it seemed to give the correct answer but I'm really not sure.</p> <p>Is it possible to compute the exact distinct element count of a huge multiset this way?</p> https://cs.stackexchange.com/questions/106475/-/106477#106477 3 Answer by Yuval Filmus for Distinct elements count of huge multiset Yuval Filmus https://cs.stackexchange.com/users/683 2019-04-04T15:20:54Z 2019-04-04T15:20:54Z <p>Let me simplify your third step: count the number of elements in your multiset which are not in the map, and add to it the number of elements in the map.</p> <p>Suppose that your elements are <span class="math-container">$x_1,\ldots,x_n$</span>. For a given hash <span class="math-container">$h$</span>, let <span class="math-container">$x_{i_1},\ldots,x_{i_\ell}$</span> be the elements hashing to <span class="math-container">$h$</span> (in order). If <span class="math-container">$\ell = 1$</span>, then the elements won't be added to the map in the first or second step. Otherwise, elements <span class="math-container">$x_{i_2},\ldots,x_{i_\ell}$</span> will be added to the map in the first step, and elements <span class="math-container">$x_{i_{\ell-1}},\ldots,x_{i_1}$</span> will be added to the map in the second step.</p> <p>In total, this shows that after the first two steps, the map will contain all elements whose hash appears more than once.</p> <p>After the third step, <em>cnt</em> will contain the number of elements whose hash appears exactly once. All of these elements are distinct, and also distinct from the elements in the map. You then add to <em>cnt</em> the number of elements present in the map, and so obtain the number of distinct elements. </p>