I am reading about the various phases of the Instruction Execution, I found out that we have three phases like below.
Now if the part I don't understand is why do we need a decode phase ? The instruction will already be stored in a binary format at some memory location, why not just fetch and execute it.
Addressing the core of your question, instructions are typically stored in a format that requires some translation (decoding) to become the control signals for instruction execution. Requiring some decoding presents an abstraction layer between the hardware and the software, allowing the control signals used for execution to be changed without breaking binary compatibility.
The instruction format exposed to software is also typically denser than the final control signals used to control execution. Typically not all possible combinations of control signals will be generated by valid instructions. For example, a very simple twos-complement adder supporting addition, subtraction, and addition with carry might have two control signals — one controlling whether one operand is inverted and the carry-in value is set to one, the other controlling whether the carry-in value is used. Both signals would be asserted for subtraction, neither for regular addition, and the second signal only for addition with carry. Asserting only the first signal would produce A plus one minus B, which is not a commonly useful operation.
In addition, naming a pipeline stage Decode is a conventional simplification. With a conventional RISC instruction encoding it is relatively easy to calculate the target of a PC-relative branch during decode because the immediate offset is a fixed field in the instruction and the PC is available. (Even if it is not known that the instruction is a taken branch early enough to allow enough time for the addition, the addition can be performed speculatively and the result ignored if the instruction is not a taken branch.) It may also be possible to evaluate branch conditions during this pipeline stage (the five-stage MIPS R2000 pipeline did this to only require a single branch delay slot to keep the pipeline full).
With a direct-mapped or way-predicted cache, the tag check can also be done in parallel with decode itself. (A cache miss or way misprediction would require the instruction to effectively be converted into a nop, but supporting modestly greater instruction cache latency might allow tradeoffs in the cache design.)
The Decode pipeline stage can also be used to detect various data and structural hazards relative to previous instructions and take mitigating actions (e.g., using result forwarding) or restraining actions (e.g., stalling the pipeline).
In a simple RISC pipeline, this stage also typically includes the reading of register operands (fixed register name positions make this independent of decode; this may involve unnecessary reads when the instruction only has one register operand but excess reads do not effect correctness). (In fact, for the 8-stage MIPS R4400 pipeline, the stage which decodes the instruction is named "Register Fetch". This stage decodes the instruction, detects interlock conditions, confirms a cache hit, and fetches register operands.)
(Such parallelism in instruction processing is not the only way that processing done during a pipeline stage may not be limited to the activity for which it is named. The division the sequential activities into pipeline stages does not have to map exactly to common conceptual divisions. For example, some of the latency of instruction cache access might be moved into the Decode stage (such that actual decoding cannot begin until some time into that stage). Similarly, some of the work associated with execution (e.g., the inversion of an operand for subtraction or the shifting of an operand by a small constant) might be hoisted into the Decode stage.)
Some processors store partially decoded instructions in the instruction cache. This can reduce the time (and potentially energy) required in the later decode at the cost of additional bits in the cache. Recent high-performance Intel and AMD x86 processors include a µop cache which stores essentially fully decoded operations, saving both time and energy and facilitating the providing of more µops per cycle.
The name "decoding" is actually a bit misleading because more things take place in this stage. You can view it as a translation from the external, programmer-visible representation of the instruction, to the CPU's internal interpretation.
All CPU architecture have an encoding of instructions, usually involving the specification of a number of op-codes specifying the "action" of the instruction (as opposed to the arguments, which might be registers, memory locations etc.). Note that the op-code is just a number, often just a byte. For instance, the addition instruction might have byte value 0x01, the subtraction might be 0x02 and so on. However, note that the numbering is somewhat arbitrary. It doesn't in itself say what the instruction does - this is something you would have to look up in the instruction manual.
What the decoding stage does is to map these numbers to the actual action of the instruction. This often involves looking up the OP-code in a data-structure, for instance a ROM. You could compare the decode stage to you looking up the op-code in a manual, to see what the instruction actually is. Now, your instruction manual is written in a human-readable language. What is the computer's manual written in? One method used in many implementations is that the ROM maps the op-code to a value which specifies the control signals that must be sent further into the execution pipeline to cause the action that is to be executed. For instance, if the instruction involves looking up the values of registers, adding them, and storing the result in another register, the ROM will contain values for the control signals that commands the register banks into read mode and the ALU into addition mode. The details of what the control signals are, are of course implementation dependent, depending on how the CPU internally functions and what control signals its various internal units accepts. However, the point is that the decode stage maps the representation of the instruction (in the form of op-code) to the actual action, in the form of these control signals. Note that on many architectures, one instruction might actually be decoded into several such "micro-instructions" with each micro-instruction essentially being a long word of all these control signals. This allows the architecture to provide the application programmer with complex instructions with short encoding. This generating and sequencing of micro-instructions might also take place in the decode stage.
Now you could ask, why not just let the instruction op-code contain all these control signals? In that case, the decode stage might not be necessary. Only problem is, that execution of an instruction often involves a lot of control signals, meaning instructions would be very long, just got the op-code. Also, it would expose the internal workings of the CPU since the external representation of instructions would now be deeply tied to the internal workings.
By the way, some architecture have very complicated encoding scheme so here the decoding involves a lot more work than just a lookup in a ROM, but in fact rather complicated logic just to determine the length and the prefixes etc. of the instruction. Also, various tricks might be necessary to keep the ROM at an acceptable size and also to allow fast look-up. On superscalar instructions, it is even possible to do this for several instructions in parallel even for encodings with variable length instructions! So as you can see, decoders can be pretty busy!
As you say, the instruction is in binary format, say 100100111001110000000111111010101000000000 (I did not actually check the number of bits). Can you execute it ?
Well, not so easy, if you do not have somewhere a description of the instruction set, including the CPU architecture, and how it is encoded in binary. You have to understand how this sequence of bits is organized, what specifies the instruction, what are the arguments or registers involved, and that may depend on the instruction. Once you have sorted that information, you pass control over to the specialized circuitry that is supposed to actually perform the instruction, say an addition, with the data or registers you have decoded. For a different instruction, it will pass control (and data) to a different circuitry.
It is like team games. There often is one guy whose role is to dispatch the ball to the most appropriate player, after decoding what the next step of the game should be. Well, this is a bit far fetched. In the CPU, there is a specific circuitry that does it: it decodes instructions.
Download a manual for a modern processor, for example here:
http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html
Look at the instruction format. And then tell me with a straight face that you can just execute it. Just for fun, try to write a function that is given an array of bytes containing an instruction for in ia64 processor in 64 bit mode, and returns the length of the instruction in bytes. Enjoy the challenge.
But the real reason that the decoder is needed: Instruction sets have been designed years in the past, and as years go by, what seemed a good idea years ago isn't a good idea today. In a typical modern processor, the instructions as described in an instruction set manual are translated into quite different instructions that the processor actually executes. Some instructions are translated into pairs of processor instructions. Sometimes multiple instructions are combined into a single processor instruction. Sometimes instructions don't get executed at all (but the instruction decoder keeps track of their effect and adjusts following instructions).
This allows the maker of the processor to change the internal instruction set, without affecting any programs written for that processor.
