Whats the point of caching if the minimum single clock cycle time is the prorogation delay of the slowest component (fetching from DRAM)?

Question

I know that the clock speed is determined by the slowest stage within the processor (usually fetch) because one clock cycle will take as much time as the slowest pipeline stage to ensure everything is synchronized.

So as fetching from DRAM is usually the slowest stage (for example during LDR instruction to memory which isn't in the cache) the time it takes for this should be the time for one clock cycle. So whats the point of caching?

I mean the processor has to wait anyway for the next clock cycle so even though the processor has fetched memory from cache very quickly wouldn't it have to wait anyway for the end of the clock cycle?

Answer 1

You are right that the clock speed is determined by the slowest stage.

But on most architectures, it is not true that fetching (or any kind of memory access) takes one cycle. This is an illusion provided by the CPU to make things manageable.

In practice, if your CPU tries to fetch, and the operation cannot be completed in one cycle, then it will stall (on pipe-lined processor it will introduce a "bubble"), until it gets the memory content.

Thus, if your memory takes 100 cycles to read, then the CPU will stall 100 times in the fetching stage. The same applies to read(and to some extent write) stages.

This is why caching is so important: by caching, the processor can reduce the fetch/read/write delay and maintain this illusion that it takes one cycle when really it does not. Example latencies for memory read are:

• L1 cache: 1 cycle
• L2 cache: 10 cycles
• DRAM: 100 cycles

Do not take those figures literally, it is very architecture dependent. But you can see that without caching, your CPU would take 100 cycles to fetch each instruction, this is obviously unacceptable.

Answer 2

When some stage is quite slow, it is divided up into multiple cycles. RAM access can take > 100 cycles. So the processor may do 100 accesses to cache in the time it takes to read one item from RAM.