Let's say I have a file which is exactly 1 MB in size.
Contents of file are :
0001001000... upto 128 bytes (128 bytes per line in file).
If I sort it in ascending or descending order (using notepad++):
Line 1: 0000000....
Line 2: 0000001.... and more.
Now if I compress it I get a 110 to 120 KB file.
I want to know if I can achieve more compression (to get a file 10 - 50 KB in size).
Is there any special compression algorithm or program for this (for sorted data) OR
I have reached the maximum compression ?
Compression program I used:
7z 19.00
Archive format: 7z
Compression level: Ultra
Compression method: LZMA2
Dictionary size: 64 KB
Word size: 64
Solid Block size: 64 MB
Number of CPU threads: 4
# Note : Whole file contains 0s and 1s only, there are no other ASCII characters.
I know that reducing the bytes per line and then compressing it will reduce the file size significantly but I don't want to do that for some reason, so reducing the bytes per line is not an option.
-
$\begingroup$ There is no magic secret to compress a file down to a certain size. $\endgroup$– Criticizing Israel not allowedCommented Sep 16, 2021 at 21:57
-
$\begingroup$ Maybe by sorting according to frequency of occurrence to assign shorter codes to higher frequencies (Huffman coding, or Hu-Tucker according to whether to keep original order) $\endgroup$– ShArCommented Sep 19, 2021 at 15:20
2 Answers
Compressibility of a file depends on the nature of the file, and what methods are applied to it. It is common for some prepressing to improve the compression.
For example, if a list of words is sorted, you will see that many words in a row with have the same prefix, and can be taken advantage to reduce the overall file size with some extra encoding. Sequential increasing integers (i.e., sorted) and taking the differences often allow one to reduce the data a lot.
Sorting is an option. If the data can be sorted without needing to be unsorted, then by all means, sort the data. Try the existing compression methods available. If you see some pattern that can be taken advantage of, and you can create a PRE/POST process that can perform the reduction, and undo the reduction to return the file back to what it was before. Great, do so.
When it comes to data compression, the more that is understood about the nature of the specific data, the more likely that some parts of it can be compressed. For example a RGB color image has the colors interleaved values, but compress poorly with generic compression, as they do not understand or attempt to do interleaving.
When I tried to delete my answer a pop up window asked if I'm voting to delete the question. I'm not voting to delete a question addressed to the whole internet world, I'm just deleting my answer
