0
$\begingroup$

I have a sequence of bytes: https://drive.google.com/file/d/17sfchPgsySi2ilIxLuBb1q-UUqq5lO87

What the data is, is unknown (see NOTE below).

I'm pretty sure this data is compressed in some way, due to the ent analysis results:

$ ent first-chunk

Entropy = 7.997831 bits per byte.

Optimum compression would reduce the size
of this 949674 byte file by 0 percent.

Chi square distribution for 949674 samples is 2962.74, and randomly
would exceed this value less than 0.01 percent of the times.

Arithmetic mean value of data bytes is 127.3922 (127.5 = random).
Monte Carlo value for Pi is 3.157487727 (error 0.51 percent).
Serial correlation coefficient is 0.001738 (totally uncorrelated = 0.0).

According to the Chi square distribution the data is definitely not random. And compression is at an optimum. This leads me to believe the data is compressed and not encrypted. Is this a correct inference?

I've tried to decompress this data assuming the compression method to be zip, gz, xz, lz4, deflate, lzma, bzip, using tools like unlzma, gunzip, unzip, unlz4, zlib-flate etc. and none of them have worked. I always end up with a "file format not recognized" or a "invalid header check" error.

How do I find what compression method was used on the data? Could it be that the compression headers are missing?

NOTE:

This data is part of a slightly larger data file that is (possibly) the firmware and additional memory of a bluetooth speaker Flash memory. The whole file is: https://drive.google.com/file/d/1e9yG8xMkZ331C2TOTOzo9Y93L4abxX9F

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ This interesting puzzle seems off-topic here. $\endgroup$
    – Yuval Filmus
    May 6, 2021 at 6:19

2 Answers 2

Reset to default
3
$\begingroup$

This file contains some repeated long byte sequences. In particular, the 66(decimal)-byte sequence beginning 49 97 CE... occurs many times, and always at an offset that's a multiple of 66.

If you decrypt the file as a bitwise Vigenère cipher with that as the key, the result has a lot of structure. It appears to be divided into 66-byte records with 64 bytes of payload followed by a 2-byte checksum. (I assume it's a checksum because it's a deterministic function of the first 64 bytes, and the first 64 bytes sometimes have simple patterns like 01 01 01... but the last two bytes always look like garbage.)

Beyond that, I can't figure anything out. Notably, while some records consist entirely of recognizable patterns (except the checksum), and some begin with recognizable patterns and end as garbage, none begin as garbage and end recognizably, which makes me think that there may be more to the encryption than a simple xor.

Improve this answer
$\endgroup$
5
  • $\begingroup$ That’s a great pointer! This means the data is encrypted and not compressed as i’d assumed? I also looked at the vigenere cipher. I don’t get why you’d use that repeating pattern as the key though, surely the key must be something else? Or did you do that just to bring out the structure? $\endgroup$
    – Capstone
    May 6, 2021 at 5:12
  • $\begingroup$ @Capstone That's the correct key iff the most common 66-byte record is all zeros. The fact that there are also records that are all 01 (except the last two bytes) when decrypted with this key is pretty strong evidence that this is the correct key and cipher (except possibly for the last two bytes). $\endgroup$
    – benrg
    May 6, 2021 at 6:15
  • $\begingroup$ ah I see. That makes sense. $\endgroup$
    – Capstone
    May 6, 2021 at 6:23
  • $\begingroup$ I'm going to try and decipher this as you did and look at the data. thanks! I'm not sure I fully understand how to decipher but I'll try first and then if I can't figure out maybe you could point me further. $\endgroup$
    – Capstone
    May 6, 2021 at 6:25
  • $\begingroup$ I was able to do a bitwise deciphering and I see now what you mean, but i still can't make any sense of the data. $\endgroup$
    – Capstone
    May 6, 2021 at 15:09
0
$\begingroup$

It's a reasonable hypothesis, but definitely not guaranteed, that it was compressed. There is general no way to know for sure which compression method was used. You've already tried some reasonable things to try.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.