I'm writing deduplication program that implements content-defined chunking. Just now i know 3 algorithms to do it (hashing a fixed-size window and selecting hashes < maxuint/N, hashing an order-1 bounded window, and local maxima chunking). Are you know other algorithms or ideas to improve these ones? My goal is to get best compression for a given average chunk size.
A few deduplication-related links: