There are two different methods: Method 1 is that the page table maps a logical address to a physical address. Since two processes use the same logical addresses (for totally different things), process B using the page table set up for process A would be fatal, so that page table needs to be swapped out. That's time consuming and painful.
Method 2 is that the page table maps a combination of logical address + an identifier for the process to a physical address. So having entries for other processes in your page table is not a problem, they will not be used, they may be swapped out at some point if the new process needs more page table address, but the transition is smooth, and there is only one method for changing page table entries.
With Method 2, in the not uncommon case that two processes don't actually need the whole page table, it's quite possible that the CPU can switch forth and back between these two processes without touching the page table at all.
No idea why this question has come back, but since it's here, a little addition. There are two things that you never, ever, ever swap out. One is your hard disk driver, because it's no good to you if it is somewhere on your hard disk and you can't read it because you need a hard disk driver in real accessible RAM. The other is the part of the page table that tells you were your page table is stored. If that is swapped out, then you can't read your page table.