I'm writing deduplication program that implements content-defined chunking. Just now i know 3 algorithms to do it (hashing a fixed-size window and selecting hashes < maxuint/N, hashing an order-1 bounded window, and local maxima chunking). Are you know other algorithms or ideas to improve these ones? My goal is to get best compression for a given average chunk size.

A few deduplication-related links:

  • $\begingroup$ Can you provide more background? I can't understand the question. $\endgroup$ Jun 26, 2013 at 19:42
  • $\begingroup$ i think it's just because it's rather narrow area. the most famous software employing deduplication is probably rsync. nowadays much faster algorithms with content-defined chunking (CDC) are used. ideally, i want to understand how to construct better CDC algorithms in order to improve compression ratio. but this probably needs either specialist in these algos or scientist who are interested to learn these area $\endgroup$
    – Bulat
    Jun 26, 2013 at 19:58
  • $\begingroup$ I think the entire term should be deprecated, if not banished as a dysphemism pushed by marketing of a company that shall go unmentioned for this very reason. rsync is not about detecting circumstantial multiplication and attempting to reduce the ensuing increase in cost: it is about keeping in check the cost of keeping intentional copies synchronised. $\endgroup$
    – greybeard
    Mar 13, 2015 at 3:11


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.