Branch delay slots in MIPS architecture

I am dealing with a standard MIPS architecture. If I have a branch instruction, for instance, beq, I know the results of the comparison in execute. However, the branching logic is actually in memory stage. Hence, doesn't that mean that I am technically losing 3 cycles(if the branch is taken) since the correct instruction is fetched during the writeback stage of the branch instruction? See the diagram below to see what I mean:

beq ...
...
label: sub

C1 C2 C3 C4 C5
beq F  D  X  M  W