3
$\begingroup$

Short version

The question & answers relate to the following topics:

  • File as a collection of records vs. file as a stream of bytes
  • Sequential File
  • Distinction between file data and metadata
  • Transfer of metadata along with the file
  • File size measurement

Original question

According to William Stallings' Operating Systems, the most common structure for a file is the sequential file, which consists of a collection of records. All files have the same amount of records and each record has a predetermined length. This seems very inefficient and doesn't match my previous understanding of files at all.

For example: if I have a JPG file, it would have records indicating things like resolution, creation date, etc. Some record fields would not be relevant for a JPG file and would be empty, but space would be reserved for those fields regardless. Since the actual data of the JPG file would be too large for any record, there would be an "overflow pointer" to where the data actually is.

Stallings also presents an alternative to file-as-a-collection-of-records: file as a stream of bytes. While the book claims it's rare, it is more in line with my understanding of files: a file is just raw data with no universal structure. However, if I move a file from one computer to another, things like "creation date" move along with it, indicating that files have some attributes which are saved within the file (as opposed to in the file management system). If all files have some header information like creation date, and that information is saved within the file, why is it that my Ubuntu claims an empty text file takes 0 bytes space?

I'm guessing that a typical file is simply a stream of bytes with no predetermined structure and that header information like the creation date is somehow transferred along with the file when it is moved. I'm guessing that the chapter about sequential files in the book is some kind of remnant from the 80's. Please correct any misconceptions that I have.

$\endgroup$
  • $\begingroup$ You're confusing file structure and file metadata. The metadata is part of the filesystem. $\endgroup$ – Yuval Filmus Apr 12 '16 at 22:50
2
$\begingroup$

I've never heard of sequential files, but apparently the term can mean two things:

  • Data which can only be accessed sequentially, for example the contents of a backup tape.
  • Record-based files (what Stallings refers to). Apparently this is still used in databases (see for example this COBOL page), but not on your desktop.

File names and other metadata such as the creation date is stored as part of the filesystem. An empty file doesn't really "waste" only zero bytes, since the metadata is stored somewhere. When you move data from filesystem to filesystem, an attempt will be made to copy the metadata, but how successful this attempt is depends on the filesystem. For example, when moving a file from one server to another, the user and group attributes may be lost, since a given user or group can exist on one server but not the other one.

$\endgroup$
  • $\begingroup$ Experts are welcome to add better answers. $\endgroup$ – Yuval Filmus Apr 12 '16 at 22:55
6
$\begingroup$

First, let me state that you may have things to unlearn. Files that are a stream of bytes are very much the rule, and have been for a long time. Almost all filesystems store a sequence of bytes, and leave the interpretation of that sequence of bytes up to applications or at least to higher-level libraries. Sequential files, consisting of a sequence (typically of variable length) of fixed-size records, went out of fashion in the 1980s if not in the 1970s. They still exist in big iron¹ (running VMS or “bigger” systems such as IBM mainframe OSes), but most programmers can go through their whole career without seeing one of those. Any book teaching sequential files as common is good only to teach operating system history, not operating systems.

Unix sees all files as byte streams. Simplifying operating system interfaces was one of Unix's major contributions, and that one stuck. Windows also sees all files as byte streams. Between them, that's essentially the whole server ecosystem, the whole workstation ecosystem, the whole mobile ecosystem², and a large chunk of the embedded ecosystem. Small embedded operating systems may have structured files, but if so I think they're generally files with a fixed structure (a single record, if you like, with set of possible formats imposed by the system) rather than a sequence of records.

In a sequential files, all the record play equivalent roles. There isn't a record for different kinds of data, but a record for different elements in a list. You can see an example in the VMS documentation. In modern terms, a sequential file is a table, a record is a row, and tables are stored in databases, not in filesystems. Although filesystems and databases are conceptually related, databases are focused on data with structure (and relations between data elements), while filesystems are focused on independent files. These approaches have different desirable security and stability guarantees, different interfaces, different performance goals, etc.

Things like the creation date³, the access permissions, etc. are generically called metadata. They are also stored in the filesystem, but outside the file's main data area. Operating systems generally report only the size of the “main” data when they report a file's size: the size isn't how much space the file occupies on the disk, but how much data is in the file. Accounting for disk usage completely accurately is rather complex — have a look at this, and keep in mind that this is only on one specific operating system.

A basic way to store files is to have every directory entry be a fixed-size structure containing the file name, other end-user metadata (timestamps, ownership information, access permissions, etc.), and some internal metadata that lets the operating system know where on the disk the file data is. A common extension to this approach is to create a level of indirection, where each directory entry contains just the file name and a pointer to another location (called an inode in the Unix world) containing the rest of the metadata.

The technical term for having multiple parts in a file with different roles (the data, the access control list, the timestamps, etc.) is a fork or stream (beware that both of these words also have different meanings). Many filesystems support only a single stream per file (the data stream), plus some fixed-size metadata. The metadata can be viewed as a stream from a conceptual point of view, but there's often no way to access it as a stream of bytes, only through dedicated functions. Some examples of uses of alternate streams are the resource fork on Mac OS, and extended (variable-size) security attributes on Linux.

The reason many file copy tools copy metadata together with the file content is that they're programmed to do so, i.e. rather than “open existing file, create new file, read data from existing file, write data to new file”, the copy algorithm also includes steps such as “read existing file's permissions, set new file's permissions to those, read existing file's timestamp, set new file's timestamp to that”.

¹ Mainframes — or mainframes emulated on x86 hardware, but running a 20-year-old emulator in which a 40-year-old operating system is running a 50-year-old application written in COBOL that nobody dares touch!
² Maybe add Symbian — where files are byte streams too.

$\endgroup$
  • $\begingroup$ Thank you for this very thorough answer. This (and your link on file size reporting) cleared many things I was wondering. I marked the other answer as "accepted", because it provides short answers to the main questions for anyone else confused by this topic. $\endgroup$ – Atte Juvonen Apr 13 '16 at 11:04
1
$\begingroup$

A little bit different view here. What a file is, is very loosely defined. There are several levels at which files mean different things.

  1. Inside file-system. From file-system perspective files can be wildly different things. For example, SunRPC Networking File System (a.k.a. NFS) sees files as messages sent on network, which have some representation on server and clients. OverlayFS (file-system used in Docker containers) sees files as collections of layers made of some read-only and some writable streams. File-system understanding of files may differ based on what storage is used (if at all), whether deduplication, compression or some other common techniques are used and so on. Not to mention physical location of data describing the files.
  2. In operating system kernel files must receive their properties, which kernel then exposes to user-space programs. Kernel functions may present same files differently to different user-space programs, depending on access permissions for example.
  3. User-space programs will often create their own presentation of files. It is customary to associate icons with files, or information such as what program is registered to deal with files.

So, depending on your view of what a file is, you will get different answers as to what its size or metadata are.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.