So I have been given the following problem and since I have no experience at all in this field, I would really appreciate some advice/guidance.
These are the requirements:
So three times per second a certain amount (fixed per session) of 32-bit values come in. These values have to be stored into a file with a timestamp per data-entry and it must be possible to at any time, with a given timestamp and interval, read out the values that are stored in that file.
This is all relatively simple to do, but this produces a 5GB file every 24 hours. If this file is zipped, it just leaves a 500MB file, so it is compressable but in that case you can't instantly retreive datapoints because you would have to unzip the entire file.
So I was thinking that this could be solved like this:
- Buffer the first 10MB of sensor data.
- Then sort all the values by the amount of times that they contain the same value. (Sensor data, so quite a few of the channels will mostly contain stable values)
- Run an algorithm that detects recurring patterns, replaces those patterns with shorter values and defines a table with the recurring patterns and their replacements.
- The compress the buffer and save it to the disk and compress every new series of sensor data with the pre-defined table.
If this would be implemented properly, it should be possible to uncompress an individual datapoint and the file is sorted by timestamp so finding the right one should be trivial with a binary search.
So I guess the questions I'm having are the follwing
- Is this at all possible? I do not expect compression results near Zip and 2x - 5x would be enough.
- What is a good (and if possible semi-easy to implement) algorithm to find patterns in binary data? I have been doing some research and I mostly ran into algorithms that only apply to text data.
Any form of help is greatly appreciated!