Why this happens. There are actually two different effects happening here:
Each file compressed independently. Some archive programs -- including zip -- compress each file independently, with no memory from one file to another file. In other words, each file is separately compressed, then the compressed files are concatenated into an archive.
Short-term memory. Some archive programs can use information about one file to help compress the next file better. They effectively concatenate the files, then compress the result. This is an improvement.
See also Nayuki's answer for more discussion of this.
However, there's a second problem. Some compression schemes -- including zip, gzip, and bzip2 -- have a limited memory. They compress the data on-the-fly, and remember the past 32KB of data, but they don't remember anything about data that occurred much earlier in the file. In other words, they can't find duplicated data if the duplicates occur farther than 32KB apart. As a result, if the identical files are short (shorter than about 32KB), the compression algorithm can remove the duplicated data, but if the identical files are long, the compression algorithm gets hosed and becomes worthless: it can't detect any of the duplicate in your data. (Bzip remembers the past 900KB or so of data, instead of 32KB.)
All standard compression algorithms have some maximum memory size, beyond which they fail to detect patterns... but for some, this number is much larger than others. For Bzip, it's something like 900KB. For xz, it's something like 8MB (with default settings). For 7z, it's something like 2GB. 2GB is more than large enough to recognize the duplicated copies of PNG files (which are typically far smaller than 2GB). Additionally, 7z also tries to be clever about placing files that are likely to be similar to each other next to each other in the archive, to help the compressor work better; tar doesn't know anything about that.
See also Raphael's answer and Nayuki's answer for more explanation of this effect.
How this applies to your setting. For your specific example, you are working with PNG images. PNG images are themselves compressed, so you can think of each PNG file as basically a sequence of random-looking bytes, with no patterns or duplication within the file. There's nothing for a compressor to exploit, if it looks at a single PNG image. Thus, if you try to compress a single PNG file (or create a zip/tar/... archive containing just a single PNG file), you won't get any compression.
Now let's look at what happens if you try to store multiple copies of the same PNG file:
Small files. If the PNG file is very small, then everything except for zip will work great. Zip will fail spectacularly: it compresses each file independently, so it has no chance to detect the redundancy/duplication among the files. Moreover, as it tries to compress each PNG file, it achieves no compression; the size of a zip archive will be huge. In contrast, the size of a tar archive (whether compressed with gzip, bzip2, or xz) and a 7z archive will be small, as it basically stores one copy of the file and then notices that the others are all identical -- they benefit from retaining memory from one file to another.
Large files. If the PNG file is large, then only 7z works well. In particular, zip continues to fail spectacularly. Also, tar.zip and tar.bzip2 fail badly, as the size of the file is larger than the compressor's memory window: as the compressor sees the first copy of the file, it can't shrink it (since it has already been compressed); by the time it starts to see the beginning of the second copy of the file, it has already forgotten the byte sequences seen at the beginning of the first file and can't make the connection that this data is actually a duplicate.
In contrast, tar.xz and 7z continue to do great with multiple copies of a large PNG file. They don't have the "small memory size" limitation and are able to notice that the second copy of the file is identical to the first copy, so there's no need to store it a second time.
What you can do about this. Use 7z. It has a bunch of heuristics that will help detect identical or similar files and compress really well in that case. You can also look at lrzip with lzop compression.
How do I know? I was able to verify this by trying some experiments with 100 copies of a file containing random bytes. I tried 100 copies of a 4KB file, 100 copies of a 1MB file, and 100 copies of a 16MB file. Here's what I found:
Size of file Size of compressed archive (with 100 copies)
zip tar.gz tar.bz2 tar.xz 7z
4KB 414KB 8KB 10KB 5KB 5KB
1MB 101MB 101MB 101MB 1MB 2MB
16MB 1.6G 1.6GB 1.6GB 1.6GB 401MB
As you can see, zip is horrible no matter how small your file is. 7z and xz are both good if your images aren't too large (but xz will be fragile and dependent on the order in which images get placed in the archive, if you have some duplicates and some non-duplicates mixed together). 7z is pretty darn good, even for large files.
References. This is also explained well in a bunch of posts over at Super User. Take a look:
- Why doesn't Gzip compression eliminate duplicate chunks of data?
- Why is Zip able to compress single file smaller than multiple files with the same content?
- Compressing many similar big files
- Compress Similar Files Efficiently
- Is there a compression or archiver program for Windows that also does deduplication?
- More efficient file compression program for many identical files?
