When you have special datasets you use special algorithms, not multipurpose tools.
The answer is that your chosen lossless compressions aren't made for what you do. Noone expects you to compress the same image twice, and even if you do it (by accident) checking against all previous input would make your algorithm O(n^2) (maybe a bit better, but the naiv approach atleast would be n^2).
Most of your compression programms you tested on run in O(n), they emphazise speed over optimal compression ratio. Noone wants to run his computer for 5 hours just to spare a few mb's, especially these days. For larger inputs anything above O(n) becomes an issue of runtime.
Another issue is ram. You can't access every part of your input at any point in time, when the input gets big enough. Even disregarding this, most people don't want to give up their whole ram or cpu just to compress something.
If you have patterns in your files that you want to compress, you will have to do manuel operations on them, write your own compression or potentially use an "archive"-type-compression (nano). A compression for longterm storage, that is totoo slow for everyday use.
Another option potentially would be a lossless video compression.