1. (A summary of the answer-in-comments by Jukka)
Startup code is stored in ROM (non-volatile memory) at a fixed address X.
When you activate the "reset" signal in the CPU, the CPU will initialise its registers to certain hard-coded values. The instruction pointer is one of these registers that are initialised to certain hard-coded values. In essence, the CPU manufacturer picks this hard-coded address X.
Hence, when you reset your CPU, the instruction pointer is reset to the same fixed address X. The CPU starts to execute instructions as usual: fetch the next instruction from address X, decode it, execute it, etc
Then whoever puts together a complete computer has to make sure that there is something useful at address X (e.g., a ROM memory chip with meaningful startup code).
2. Elaboration
The CPU has a very simple way of working, known as the FETCH-EXECUTE cycle:
(a) FETCH: It goes to the memory and brings the instruction that
is at address [PC]. (PC is a name of an internal CPU register; it
holds the address of the instruction we need to perform next..).
(b) Execute: The PC executes the instruction it has just
retrieved from the memory
(c) PC $\gets$ PC+1.(*)
(d) Go back to (a).
This fetch-execute loop happens all the time, regardless of the program you are running. In particular, when the CPU boots up (say, when its power goes from 0 to Vcc) the CPU simply begins to perform the above loop.
The initial value of PC depends on the specific CPU. It can be that the CPU wakes up with PC=0, or it can be that PC=0xFFF0 (as happens with Intel's X86 family), or with any other value PC=$PC_{init}$.
If you press the reset button, or turn the power off and on, the CPU restores the PC to its initial value $PC_{init}$ and the CPU begins running the program that is in the memory at address $PC_{init}$, through the fetch-execute loop.
Now it is the computer designer task to make sure that the BIOS sits in the memory at address $PC_{init}$. The BIOS usually resides in a non-volatile memory so there is always a program to run in that address, even if you just turned the power off and on (this deletes the RAM but not the BIOS. It is more difficult to delete/change the BIOS, although possible and happens, e.g., when your device updates its "firmware").
(*) Advancing the PC by +1 assumes that each instruction takes 1 address of the memory space. The is rarely the case. In real systems we may see here PC+4 (as in the 32b MIPS) or PC+x (with $x\in\{1,...,7\}$ which depends on the instruction itself) as in the X86 family, etc.
