I want to write a compression algorithm for a specific use case that I have. I have many arrays which are for the most part, very similar. Each value in the array is an integer and is related to the corresponding value in the previous array.

These arrays have a correlation coefficient of around 0.8 and they will be stored together in the same file in a binary format

How do I go about writing an algorithm to losslessly compress these arrays for minimum storage size?

  • 2
    $\begingroup$ I don't think the question is answerable - I don't think you've given us enough information to suggest an algorithm. In particular, it is not clear what notion of 'similar' or 'related' might apply here, and without specific understanding of those details, I doubt it will be possible to design a concrete algorithm. Many of the details that are provided (e.g., binary format, stored together in the same file) are an implementation detail that seem unimportant to algorithm design. $\endgroup$
    – D.W.
    Commented Apr 5, 2022 at 4:56
  • 1
    $\begingroup$ Did you try to convert every part but the first to a "delta" and compare result of some implementations on original and converted data? Next would be to transpose the data. Results? $\endgroup$
    – greybeard
    Commented Apr 5, 2022 at 7:09
  • $\begingroup$ A correlation of 0.8 is not that spectacular. I think you'd better explain us why these arrays are similar. $\endgroup$
    – user16034
    Commented Apr 5, 2022 at 15:36


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.