High level reasons
When you think about it, a microprocessor does an amazing thing: it lets you take a machine (such as a washing machine or an elevator), and replace a whole chunk of custom-designed mechanisms or circuits with a cheap, mass-produced silicon chip. You save a lot of money on parts, and a lot of time on design.
But hang on, a standard chip, replacing countless custom designs? There can't be a single, perfect microprocessor that is perfect for every application. Some applications need to minimise power usage but don't need to be fast; others need to be fast but don't need to be easy to program, others need to be low cost, etc.
So, we have many different "flavours" of microprocessor, each with its own strengths and weaknesses. It is desirable for them to all use a compatible instruction set, because this allows code reuse and makes it easier to find people with the right skills. However, the instruction set does affect the cost, complexity, speed, ease-of-use, and physical constraints of the processor, and so we have a compromise: there a few "mainstream" instruction sets (and many minor ones), and within each instruction set there are many processors with different characteristics.
Oh, and as technology changes, all these trade-offs change, so instruction sets evolve, new ones emerge, and old ones die. Even if there was a "best" instruction set of today, it might not be in 20 years.
Probably the biggest design decision in an instruction set is the word size, i.e. how large a number the processor can "naturally" manipulate. 8-bit processors deal with numbers from 0-255, whereas 32-bit processors deal with numbers from 0 to 4,294,967,295. Code designed for one needs to be completely rethought for another.
It is not just a matter of translating instructions from one instruction set to another. A completely different approach may be preferable in a different instruction set. For example, on an 8-bit processor a lookup table may be ideal, while on a 32-bit processor an arithmetic operation would be better for the same purpose.
There are other major differences between instruction sets. Most instructions fall into four categories:
- Computation (Arithmetic and logic)
- Control Flow
- Data Transfer
- Processor configuration
Processors differ in what sort of computations they can perform, as well as how they approach control flow, data transfer, and processor configuration.
For example, some AVR processors can neither multiply nor divide; whereas all x86 processors can. As you may imagine, eliminating the circuitry required for tasks like multiplication and division can make a processor simpler and cheaper; these operations can still be performed using software routines if they are needed.
x86 allows arithmetic instructions to load their operands from memory and/or save their results to memory; ARM is a load-store architecture and thus only has a few dedicated instructions for accessing memory. Meanwhile x86 has dedicated conditional-branch instructions, while ARM allows practically all instructions to be conditionally executed. Also, ARM allows bit-shifts to be performed as part of most arithmetic instructions. These differences lead to different performance characteristics, differences in internal design and cost of the chips, and differences in programming techniques at the assembly language level.
The reason it is impossible to have a universal assembly language is that, to properly convert assembly code from one instruction set to another, one must design the code all over again—something computers cannot yet do.