The other two answers (at the time of writing) talk about interrupts and the IDT. This is correct, however, on a modern Intel-esque CPU, there are no fewer than three ways to call into a kernel.
Method #1: Interrupts.
This is explained above. You set up an entry in the interrupt descriptor table/interrupt vector, and then execute a software interrupt to enter the kernel.
The main advantage of this method is that a typical kernel needs to be able to handle interrupts anyway, and it works on archaic hardware.
Method #2: Call gates.
A call gate is a special kind of segment selector. The target of the call needs to be loaded in the global or local segment descriptor table (GDT and LDT respectively). If you then perform a far call instruction using the call gate as the segment (the offset of the call is ignored), this allows you to call more privileged code. Call gates are extremely flexible; the IA-32 architecture has four privilege levels, and call gates let you call any level.
I don't believe that Linux ever used call gates, but Windows 95 did. Win95 kernel services (krnl386.exe
and kernel.dll
) actually ran in user mode (ring 3). The highest privilege level (ring 0) was only used for drivers and a microkernel which performed only process switching. Calling into drivers was done using call gates. This allowed legacy 16-bit code (of which there was a lot!) to use Win95 drivers just using a standard far call, just like they always did.
Inadequate protection of the global descriptor table was the cause of several Windows 95 exploits, which managed to install their own call gates by writing over memory.
Method #3: SYSCALL/SYSRET and SYSENTER/SYSEXIT
These are two sets of instructions, independently invented by AMD and Intel, but they essentially do the same thing. SYSCALL/SYSRET came first and was AMD-only, SYSENTER/SYSEXIT was Intel, but AMD implements it now. So I'm going to describe SYSENTER/SYSEXIT.
Unlike call gates, SYSENTER can only be used to transfer to ring 0, and can only transfer to one location. However, it has the advantage of being extremely low-latency because unlike a call or interrupt it doesn't touch the stack.
The transfer location is set up using three model-specific registers: one for the segment information, and one each for the instruction pointer and stack pointer of the kernel code. Because nothing is "pushed" onto the stack, the user mode code is responsible for telling the kernel where to return to by passing the return instruction pointer and stack pointer in registers. The kernel is responsible for restoring the stack pointer, and the SYSEXIT instruction restores the instruction pointer.
Further information on the SYSENTER and SYSEXIT instructions.