How does the DB manage a pool of buffers when the operating system ends up controlling what's really in memory? For example, couldn't the operating system decide to evict a page frame from a DB's buffer?
Yes, absolutely, that could happen, and it makes it harder for databases to optimize their own performance.
Some databases bypass the operating system's virtual memory management and manage memory themselves. They might request that the operating system pin a range of the address space into memory (and never page it out). Or, they might be designed to be the only application that runs on the machine, and take care of their own memory management so they never use too much memory -- hoping that this will make it unlikely that the OS ever pages out any of their data.