Why XML dumps from Wikipedia are so compressed?

Question

Few years ago I downloaded an XML dump of one of Wikipedia's fandom sites.
It was the XML datadump of Marvel Fandom.
The XML was zipped , size was 900 MB (approx).
But when I unzipped it , it was somewhere 10 - 11 GB big.
And according to Wiki Taxi (A software I used to convert those XML dump into readable format for offline reading) it had 1,50,000 pages (approx).
So, I want to know is :
Do they used any specific kind of algorithm to compress it ?
How they achieved so much compression, despite having 1,50,000 pages ?
Do wikipedia uses any "more compressible" XML syntax ?
Will converting ebooks to XML be any helpful in reduction of size of ebook ?

Answer 1

There are two dumps available at https://marvel.fandom.com/wiki/Special:Statistics at the moment. The "current pages" dump is 1.6 GB uncompressed, 167 MB compressed, for a ratio of 9.5:1, and the "current pages and history" dump is 30 GB uncompressed, 552 MB compressed, for a ratio of 55:1.

Both are compressed using LZMA2, which is a standard, widely used algorithm. It's an LZ77 variant, meaning that if a string of bytes appears more than once within the window distance (here 16 MiB), all occurrences after the first are replaced with references to the first one, usually costing just a few bytes each time.

The full-history dump compresses so well because most page versions are very similar to each other, with only small parts changed. Most of the text of most versions can be compressed down to a reference to a previous version which takes just a few bytes.

The current-pages dump is highly compressible because it contains a lot of boilerplate text. For example, here's an excerpt from an image description:

| Others1     = [[Red Skull (1940s Impostor) (Earth-616)|1947's Unnamed Impostor]]; Red Skull (1940s Impostor) (Earth-616) from Captain America Comics Vol 1 61 Cover.jpg
| Others2     = [[Red Skull (1950s Impostor) (Earth-616)|1950's Unnamed Impostor]]; Red Skull (1950s Impostor) (Earth-616).jpg
| Others3     = Albert Malik (Earth-616); Albert Malik (Earth-616) from Amazing Spider-Man Vol 5 69 001.jpg
| Others4     = George Maxon (Earth-616); George Maxon (Earth-616) from Captain America Comics Vol 1 1 0002.jpg
| Others5     = [[Johann Shmidt (Heroes Reborn) (Earth-616)|Johann Shmidt<br>(Heroes Reborn)]]; Johann Shmidt (Heroes Reborn) (Earth-616).jpg
| Others6     = [[Red Skull (Tsum Tsum) (Earth-616)|Tsum Tsum]]; Red Skull (Tsum Tsum) (Earth-616) from Marvel Tsum Tsum Vol 1 4 0001.jpg
| Others7     = [[Red Skull (Earth-1610)|Captain America's Unnamed Son]]; Ultimate Comics Avengers Vol 1 1 Textless Red Skull Variant.jpg

It goes on up to Others17, and that's just part of the description, and there are probably similar descriptions on many other images. This is full of long duplicated strings that can be compressed to a few bytes after their first occurrence.

Even ordinary English text tends to compress fairly well for similar reasons.

Will converting ebooks to XML be any helpful in reduction of size of ebook ?

No. But ebooks tend to be small anyway if they're just text.