Skip to main content
replaced http://cs.stackexchange.com/ with https://cs.stackexchange.com/
Source Link
  • Each file compressed independently. Some archive programs -- including zip -- compress each file independently, with no memory from one file to another file. In other words, each file is separately compressed, then the compressed files are concatenated into an archive.

  • Short-term memory. Some archive programs can use information about one file to help compress the next file better. They effectively concatenate the files, then compress the result. This is an improvement.

    See also Nayuki's answerNayuki's answer for more discussion of this.

    However, there's a second problem. Some compression schemes -- including zip, gzip, and bzip2 -- have a limited memory. They compress the data on-the-fly, and remember the past 32KB of data, but they don't remember anything about data that occurred much earlier in the file. In other words, they can't find duplicated data if the duplicates occur farther than 32KB apart. As a result, if the identical files are short (shorter than about 32KB), the compression algorithm can remove the duplicated data, but if the identical files are long, the compression algorithm gets hosed and becomes worthless: it can't detect any of the duplicate in your data. (Bzip remembers the past 900KB or so of data, instead of 32KB.)

    All standard compression algorithms have some maximum memory size, beyond which they fail to detect patterns... but for some, this number is much larger than others. For Bzip, it's something like 900KB. For xz, it's something like 8MB (with default settings). For 7z, it's something like 2GB. 2GB is more than large enough to recognize the duplicated copies of PNG files (which are typically far smaller than 2GB). Additionally, 7z also tries to be clever about placing files that are likely to be similar to each other next to each other in the archive, to help the compressor work better; tar doesn't know anything about that.

    See also Raphael's answerRaphael's answer and Nayuki's answerNayuki's answer for more explanation of this effect.

  • Each file compressed independently. Some archive programs -- including zip -- compress each file independently, with no memory from one file to another file. In other words, each file is separately compressed, then the compressed files are concatenated into an archive.

  • Short-term memory. Some archive programs can use information about one file to help compress the next file better. They effectively concatenate the files, then compress the result. This is an improvement.

    See also Nayuki's answer for more discussion of this.

    However, there's a second problem. Some compression schemes -- including zip, gzip, and bzip2 -- have a limited memory. They compress the data on-the-fly, and remember the past 32KB of data, but they don't remember anything about data that occurred much earlier in the file. In other words, they can't find duplicated data if the duplicates occur farther than 32KB apart. As a result, if the identical files are short (shorter than about 32KB), the compression algorithm can remove the duplicated data, but if the identical files are long, the compression algorithm gets hosed and becomes worthless: it can't detect any of the duplicate in your data. (Bzip remembers the past 900KB or so of data, instead of 32KB.)

    All standard compression algorithms have some maximum memory size, beyond which they fail to detect patterns... but for some, this number is much larger than others. For Bzip, it's something like 900KB. For xz, it's something like 8MB (with default settings). For 7z, it's something like 2GB. 2GB is more than large enough to recognize the duplicated copies of PNG files (which are typically far smaller than 2GB). Additionally, 7z also tries to be clever about placing files that are likely to be similar to each other next to each other in the archive, to help the compressor work better; tar doesn't know anything about that.

    See also Raphael's answer and Nayuki's answer for more explanation of this effect.

  • Each file compressed independently. Some archive programs -- including zip -- compress each file independently, with no memory from one file to another file. In other words, each file is separately compressed, then the compressed files are concatenated into an archive.

  • Short-term memory. Some archive programs can use information about one file to help compress the next file better. They effectively concatenate the files, then compress the result. This is an improvement.

    See also Nayuki's answer for more discussion of this.

    However, there's a second problem. Some compression schemes -- including zip, gzip, and bzip2 -- have a limited memory. They compress the data on-the-fly, and remember the past 32KB of data, but they don't remember anything about data that occurred much earlier in the file. In other words, they can't find duplicated data if the duplicates occur farther than 32KB apart. As a result, if the identical files are short (shorter than about 32KB), the compression algorithm can remove the duplicated data, but if the identical files are long, the compression algorithm gets hosed and becomes worthless: it can't detect any of the duplicate in your data. (Bzip remembers the past 900KB or so of data, instead of 32KB.)

    All standard compression algorithms have some maximum memory size, beyond which they fail to detect patterns... but for some, this number is much larger than others. For Bzip, it's something like 900KB. For xz, it's something like 8MB (with default settings). For 7z, it's something like 2GB. 2GB is more than large enough to recognize the duplicated copies of PNG files (which are typically far smaller than 2GB). Additionally, 7z also tries to be clever about placing files that are likely to be similar to each other next to each other in the archive, to help the compressor work better; tar doesn't know anything about that.

    See also Raphael's answer and Nayuki's answer for more explanation of this effect.

replaced http://superuser.com/ with https://superuser.com/
Source Link
added 300 characters in body
Source Link
D.W.
  • 165.6k
  • 21
  • 230
  • 490
  • Each file compressed independently. Some archive programs -- including zip -- compress each file independently, with no memory from one file to another file. In other words, each file is separately compressed, then the compressed files are concatenated into an archive.

  • Short-term memory. Some archive programs can use information about one file to help compress the next file better. They effectively concatenate the files, then compress the result. This is an improvement.

    See also Nayuki's answer for more discussion of this.

    However, there's a second problem. Some compression schemes -- including zip, gzip, and bzip2 -- have a limited memory. They compress the data on-the-fly, and remember the past 32KB of data, but they don't remember anything about data that occurred much earlier in the file. In other words, they can't find duplicated data if the duplicates occur farther than 32KB apart. As a result, if the identical files are short (shorter than about 32KB), the compression algorithm can remove the duplicated data, but if the identical files are long, the compression algorithm gets hosed and becomes worthless: it can't detect any of the duplicate in your data. (Bzip remembers the past 900KB or so of data, instead of 32KB.)

    Others -- including xz and 7z -- don'tAll standard compression algorithms have some maximum memory size, beyond which they fail to detect patterns... but for some, this problemnumber is much larger than others. For instanceBzip, it's something like 900KB. For xz, it's something like 8MB (with default settings). For 7z has a, it's something like 2GB memory size. That's 2GB is more than large enough to recognize the duplicated copies of PNG files (which are typically far smaller than 2GB). Additionally, 7z also tries to be clever about placing files that are likely to be similar to each other next to each other in the archive, to help the compressor work better; tar doesn't know anything about that.

    See also Raphael's answer and Nayuki's answer for more explanation of this effect.

Size of file      Size of compressed archive (with 100 copies)
                  zip  tar.gz  tar.bz2  tar.xz    7z
         4KB    414KB     8KB     10KB     5KB    5KB
         1MB    101MB   101MB    101MB     1MB    2MB
        16MB    1.6G    1.6GB    1.6GB   1.6GB  401MB

As you can see, zip is horrible no matter how small your file is. 7z and xz are both good if your images aren't too large (but xz will be fragile and dependent on the order in which images get placed in the archive, if you have some duplicates and some non-duplicates mixed together). 7z is pretty darn good, even for large files.

  • Each file compressed independently. Some archive programs -- including zip -- compress each file independently, with no memory from one file to another file. In other words, each file is separately compressed, then the compressed files are concatenated into an archive.

  • Short-term memory. Some archive programs can use information about one file to help compress the next file better. They effectively concatenate the files, then compress the result. This is an improvement.

    However, there's a second problem. Some compression schemes -- including zip, gzip, and bzip2 -- have a limited memory. They compress the data on-the-fly, and remember the past 32KB of data, but they don't remember anything about data that occurred much earlier in the file. In other words, they can't find duplicated data if the duplicates occur farther than 32KB apart. As a result, if the identical files are short (shorter than about 32KB), the compression algorithm can remove the duplicated data, but if the identical files are long, the compression algorithm gets hosed and becomes worthless: it can't detect any of the duplicate in your data. (Bzip remembers the past 900KB or so of data, instead of 32KB.)

    Others -- including xz and 7z -- don't have this problem. For instance, 7z has a 2GB memory size. That's more than large enough to recognize the duplicated copies of PNG files (which are typically far smaller than 2GB). Additionally, 7z also tries to be clever about placing files that are likely to be similar to each other next to each other in the archive, to help the compressor work better; tar doesn't know anything about that.

Size of file      Size of compressed archive (with 100 copies)
                  zip  tar.gz  tar.bz2  tar.xz    7z
         4KB    414KB     8KB     10KB     5KB   5KB
         1MB    101MB   101MB    101MB     1MB   2MB
  • Each file compressed independently. Some archive programs -- including zip -- compress each file independently, with no memory from one file to another file. In other words, each file is separately compressed, then the compressed files are concatenated into an archive.

  • Short-term memory. Some archive programs can use information about one file to help compress the next file better. They effectively concatenate the files, then compress the result. This is an improvement.

    See also Nayuki's answer for more discussion of this.

    However, there's a second problem. Some compression schemes -- including zip, gzip, and bzip2 -- have a limited memory. They compress the data on-the-fly, and remember the past 32KB of data, but they don't remember anything about data that occurred much earlier in the file. In other words, they can't find duplicated data if the duplicates occur farther than 32KB apart. As a result, if the identical files are short (shorter than about 32KB), the compression algorithm can remove the duplicated data, but if the identical files are long, the compression algorithm gets hosed and becomes worthless: it can't detect any of the duplicate in your data. (Bzip remembers the past 900KB or so of data, instead of 32KB.)

    All standard compression algorithms have some maximum memory size, beyond which they fail to detect patterns... but for some, this number is much larger than others. For Bzip, it's something like 900KB. For xz, it's something like 8MB (with default settings). For 7z, it's something like 2GB. 2GB is more than large enough to recognize the duplicated copies of PNG files (which are typically far smaller than 2GB). Additionally, 7z also tries to be clever about placing files that are likely to be similar to each other next to each other in the archive, to help the compressor work better; tar doesn't know anything about that.

    See also Raphael's answer and Nayuki's answer for more explanation of this effect.

Size of file      Size of compressed archive (with 100 copies)
                  zip  tar.gz  tar.bz2  tar.xz    7z
         4KB    414KB     8KB     10KB     5KB    5KB
         1MB    101MB   101MB    101MB     1MB    2MB
        16MB    1.6G    1.6GB    1.6GB   1.6GB  401MB

As you can see, zip is horrible no matter how small your file is. 7z and xz are both good if your images aren't too large (but xz will be fragile and dependent on the order in which images get placed in the archive, if you have some duplicates and some non-duplicates mixed together). 7z is pretty darn good, even for large files.

Source Link
D.W.
  • 165.6k
  • 21
  • 230
  • 490
Loading