Understanding memory mapping conceptually

Question

I've already read several blogs and questions on stack exchange, but I'm unable to grasp what the real drawbacks of memory mapped files are. I see the following are frequently listed:

1. You can't memory map large files (>4GB) with a 32-bit address space.

QUESTION #1: Why? Isn't that the whole point of virtual memory? If a file is greater than 4GB, it may cause trashing by swapping out some memory mapped pages, but why is there a limitation?

2. If the application is trying to read from a part of the file that is not loaded in the page cache, it (the application) will incur a penalty in the form of a page-fault, which in turn means increased I/O latency for the operation.

QUESTION #2: Isn't this the case for a standard file I/O operation as well? If an application tries to read from a part of a file that is not yet cached, it will result in a syscall that will cause the kernel to load the relevant page/block from the device. And on top of that, the page needs to be copied back to the user-space buffer.

Is the concern here that page-faults are somehow more expensive than syscalls in general - my interpretation of what Linus Torvalds says here? Is it because page-faults are blocking => the thread is not scheduled off the CPU => we are wasting precious time? Or is there something I'm missing here?

3. Overhead of kernel mappings and data structures - according to Linus Torvalds. I won't even attempt to question this premise, because I don't know much about the internals of Linux kernel. :)

4. No support for async I/O for memory mapped files.

QUESTION #3: Is there an architectural limitation with supporting async I/O for memory mapped files, or is it just that it no one got around to doing it?

5. One drawback that I thought of was that if too many files are memory mapped, this can cause lower available system resources (memory) => can cause pages to be evicted => potentially more page faults. So some prudence is required in deciding what files to memory map and their access patterns.

QUESTION #4: Vaguely related, but my interpretation of this article is that the kernel can read-ahead for standard I/O (even without fadvise()) but does not read-ahead for memory mapped files (unless issued an advisory with madvice()). Is this accurate? If this statement is in-fact true, is that why syscalls for standard I/O maybe faster, as opposed to a memory mapped file which will almost always cause a page-fault?

Answer 1

With memory-mapped files, a file is mapped to a pointer p, and byte #i of the file can be accessed through the pointer p+i. For each byte in the file, there must be an address available in the logical address space. With 32 bit, there are only 2^32 possible logical addresses, so 2^32 bytes in a memory mapped file that can be accessed at all.

You are confusing address space and RAM. A 64 bit computer with just 1 GB if RAM can memory map multi-terabyte files without problems. There will be lots of paging, but it will work. With a 32 bit computer, no chance.

Answer 2

Re #1: If your address space is 32 bits, you can address only 4GiB by pointers into memory. You can process larger files by e.g. reading the sequentially, or lseek(2) into it (note that usually off_t is a 64-bit type today).

Re #2: Yes, the operating system will read (part of) the file into memory for munging it. The offset into the on-disk file doesn't have to be limited to 32 bits. For quite some time now Unix/Linux handles "large files" with 64 bit file offsets, lately by default (in the beginning some incantation was needed to enable that).