Skip to main content
2 of 2
replaced http://unix.stackexchange.com/ with https://unix.stackexchange.com/

First, let me state that you may have things to unlearn. Files that are a stream of bytes are very much the rule, and have been for a long time. Almost all filesystems store a sequence of bytes, and leave the interpretation of that sequence of bytes up to applications or at least to higher-level libraries. Sequential files, consisting of a sequence (typically of variable length) of fixed-size records, went out of fashion in the 1980s if not in the 1970s. They still exist in big iron¹ (running VMS or “bigger” systems such as IBM mainframe OSes), but most programmers can go through their whole career without seeing one of those. Any book teaching sequential files as common is good only to teach operating system history, not operating systems.

Unix sees all files as byte streams. Simplifying operating system interfaces was one of Unix's major contributions, and that one stuck. Windows also sees all files as byte streams. Between them, that's essentially the whole server ecosystem, the whole workstation ecosystem, the whole mobile ecosystem², and a large chunk of the embedded ecosystem. Small embedded operating systems may have structured files, but if so I think they're generally files with a fixed structure (a single record, if you like, with set of possible formats imposed by the system) rather than a sequence of records.

In a sequential files, all the record play equivalent roles. There isn't a record for different kinds of data, but a record for different elements in a list. You can see an example in the VMS documentation. In modern terms, a sequential file is a table, a record is a row, and tables are stored in databases, not in filesystems. Although filesystems and databases are conceptually related, databases are focused on data with structure (and relations between data elements), while filesystems are focused on independent files. These approaches have different desirable security and stability guarantees, different interfaces, different performance goals, etc.

Things like the creation date³, the access permissions, etc. are generically called metadata. They are also stored in the filesystem, but outside the file's main data area. Operating systems generally report only the size of the “main” data when they report a file's size: the size isn't how much space the file occupies on the disk, but how much data is in the file. Accounting for disk usage completely accurately is rather complex — have a look at this, and keep in mind that this is only on one specific operating system.

A basic way to store files is to have every directory entry be a fixed-size structure containing the file name, other end-user metadata (timestamps, ownership information, access permissions, etc.), and some internal metadata that lets the operating system know where on the disk the file data is. A common extension to this approach is to create a level of indirection, where each directory entry contains just the file name and a pointer to another location (called an inode in the Unix world) containing the rest of the metadata.

The technical term for having multiple parts in a file with different roles (the data, the access control list, the timestamps, etc.) is a fork or stream (beware that both of these words also have different meanings). Many filesystems support only a single stream per file (the data stream), plus some fixed-size metadata. The metadata can be viewed as a stream from a conceptual point of view, but there's often no way to access it as a stream of bytes, only through dedicated functions. Some examples of uses of alternate streams are the resource fork on Mac OS, and extended (variable-size) security attributes on Linux.

The reason many file copy tools copy metadata together with the file content is that they're programmed to do so, i.e. rather than “open existing file, create new file, read data from existing file, write data to new file”, the copy algorithm also includes steps such as “read existing file's permissions, set new file's permissions to those, read existing file's timestamp, set new file's timestamp to that”.

¹ Mainframes — or mainframes emulated on x86 hardware, but running a 20-year-old emulator in which a 40-year-old operating system is running a 50-year-old application written in COBOL that nobody dares touch!
² Maybe add Symbian — where files are byte streams too.