1
$\begingroup$

I have this large text file that when unzipped has about 2GB. I split this one into multiple(more than 5 million) files and now I have a folder of about 20GB, how is this possible?

$\endgroup$
  • $\begingroup$ I think this question is more suited to Super User since it has nothing to do with computer science, and everything with real hard- and software. Community votes, please: Offtopic? Should we migrate? $\endgroup$ – Raphael Apr 7 '17 at 20:38
  • $\begingroup$ @Raphael perhaps but the answers have implications on abstract data structures like acyclic graphs. If we really want to go out there we could say that it relates to encoding interleaved information on an stack based automaton. Then again you're probably right. $\endgroup$ – Aluan Haddad Apr 9 '17 at 23:05
  • $\begingroup$ I don't see this implication. Memory/storage alignment issues are firmly in the technology domain. $\endgroup$ – Raphael Apr 10 '17 at 8:04
8
$\begingroup$

Filesystem overhead.

Every time you create a file, the filesystem reserves space on disk to store the file name, and other metadata like its permissions, creation date, etc. It also has to maintain a sort of index of all the (possibly sparse) blocks occupied by each file. The index might be a simple list or a tree, depending on the filesystem.

Finally, note that most filesystems can not allocate only a portion of a block for a file, so the allocated space needs to be rounded up to the block size. This rounding affects both the file blocks, and the index blocks.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.