# Does there exist a data compression algorithm that uses a large dataset distributed with the encoder/decoder?

If my goal were to compress say 10,000 images and I could include a dictionary or some sort of common database that the compressed data for each image would reference, could I use a large dictionary shared by the entire catalog and therefore get much smaller file sizes? Could this be expanded to work with images in general, i.e. to replace something like JPEG?

Are there existing compression systems that operate like this, where there is a large common set of bits transmitted and loaded before decompression, that has been built by analyzing many images?

For example, is there an existing computer science/machine learning research effort using sparse autoencoding over a large set of images and this concept of distributing a network derived from that encoding with the decompressor?

Note: Coming back to this question and the answers, I see that it was primarily not understood because those that read it did not understand enough about new types of machine learning.

New techniques in machine learning, particularly deep learning or other advanced neural network systems, do have the potential make many existing data compression techniques outdated. Or rather, we should expect that some more extensive research will result in these types of powerful encoders. As of yet this is mainly speculative, and data compression scientists cannot conceive of them without fully understanding the power of the new neural network techniques.

• Some tips on improving this kind of question, for the future. 1. This site is best for technical questions. It's not so good for proposals ("I suggest such-and-such a scheme would be awesome; am I right, or what?") or expressions of opinions. I suggest that you stick to just a technical question, without expressing an opinion or point of view (especially when that opinion/point of view is not really necessary). 2. We expect you to do serious research on your own first, and tell us in the question what you've tried. So, what research have you done? What have you tried? Where have you looked? – D.W. Mar 6 '14 at 1:22
• The point of view was intended to give context for people because I had previously asked this question on Hacker News and it was not understood. You suggest that I have not done serious research. I have done research on the basics of data compression. I have not seen such an idea related to image compression. But it seems so obvious that I assumed that the experts here could easily point me towards existing research that I missed. Is there a problem with my question, or are you making an issue because you just don't know the answer? – Jason Livesay Mar 6 '14 at 2:46
• Thanks for the answer. Can you give me a hint about what to google to find the existing algorithms that work this way? – Jason Livesay Mar 6 '14 at 3:19
• Compression algorithms generally work somewhat like you say, but they collect the "dictionary" from the data to be compressed. – vonbrand Mar 7 '14 at 15:46
• Editing to ask that answers discuss what might be possible with machine learning changes the question in a fairly significant way. I'm not sure whether you'll get any answers to that particular aspect -- you might do better to ask that separately. Contrary to what you wrote, I don't think the issue is that people who read it didn't understand machine learning -- I suspect the reason you didn't get any answers addressing that is because you didn't raise it in the question as something you'd like to see considered. – D.W. Jun 22 '16 at 7:23

you're focused on images so will construct a scenario that fits your question in the affirmative. imagine certain images that are not compressed via lossy compression eg a non jpeg-like algorithm. imagine the images are not compressed and stored in a straight pixel format. next suppose there are images that are composed of sub images in a vertical fashion. eg imagine a stack of images composing a larger image:

A
B
C
D


in the above crude diagram A,B,C,D are stacked images that compose a larger image, say E. now suppose that one creates a file that is concatenating files/images A,B,C,D,E. on a standard "lossless" dictionary compression algorithm like lempel-ziv, the algorithm will be smart enough to use A,B,C,D as long dictionary strings, and compose E out of those strings. (this depends on specifics like maximal dictionary word size etc, but it could be made to work). so in this admittedly rather contrived scenario, the answer is YES.

however, you are missing something about JPEG, it is typically lossy compression, and also images are generally a 2d format, whereas a lot of compression algorithms take advantage of 1d patterns like long strings. so this lossiness combined with the 2d format will mean that general compression algorithms will not really save much over the cost to store the original images (which generally are already compressed anyway).

therefore this is generally a negative answer overall, the overall idea of "compressing" using a "dictionary of images" is not really valid/possible/plausible and you can get a better idea of why by studying the mechanics/dynamics of compression algorithms in a more detailed way. on the other hand there are a wide variety of image compression algorithms that do in fact take advantage of redundancies in images. in JPEG for example the image is broken into squares and it can reuse common squares!

• OK, but not even with sparse recursive autoencoding? youtube.com/watch?v=n1ViNeWhC24 – Jason Livesay Apr 2 '14 at 0:32
• "imagine certain images that are not compressed via lossy compression eg a non jpeg-like algorithm." You mean images that are just not compressed? Or images that are compressed but losslessly? Or soemthing else? Please write more clearly. – David Richerby Aug 19 '14 at 12:10
• jpeg-like algorithms are generally lossy. ie compressed but not using a lossy algorithm. – vzn Aug 19 '14 at 15:06