When you try to access data from a user space program, what happens is that the hardware tries to translate your logical address into the physical address of RAM, and then access the data if possible. If the data is not in RAM, then the CPU will switch to code in the OS that tries to load the data into RAM.
Mostly this is possible for code in the kernel as well. However, there is some data that absolutely MUST be in RAM: That is the code for loading virtual memory from disk to the RAM, and all the data that keeps track of what is where in memory. If the code loading data from disk into RAM is on the disk and not in RAM, then obviously you can't execute it.
Many CPUs will produce a fatal error in that situation.