# Implementation of caches on CPUs with pipelines

I've read that some current CPUs (e.g. Intel i7 and ARM A9) have (L1) cache latencies of multiple clock cycles while also being pipelined. Some devote multiple pipeline stages to instruction fetching.

How is the L1 cache implemented to support pipelining?

Is it, say, $k$ actual stages for decoding the address or does it use some asynchronous multiport logic with a datapath latency set to $k$ clock cycles?

• Are you asking about the number of stages in a cache access (your claimed i7 number) or the number of stages in an instruction fetch (your claimed a9 number)? Instruction fetch involves a great deal more than a cache access. And a "cache access" may or may not include the cost of doing a virtual to physical translation. – Wandering Logic Sep 2 '14 at 11:13
• What are your thoughts? Where do you see particular challenges? – Raphael Sep 2 '14 at 11:25
• I couldn't find any mention of hit time for the Arm A9 so I made an assumption that the 3 stages were for a L1 hit (common case). Another assumption I'm making in my question is that the latency is the complete fetch operation for a successful address translation and cache hit. If I'm wrong can you explain what really happens. – 102948239408 Sep 2 '14 at 12:17
• csl.cornell.edu/courses/ece4750/handouts/… Looking at the slides here, there is a two cycle hit latency on the instruction cache, yet none of the microarchitecture examples of the cache implementations, the only pipelined cache is for writes. As the cache in question is instruction which won't recieve any write transactions. Why and how is the cache implemented so that there is a 2 cycle latency as the tag compare can happen in parallel to decoding. – 102948239408 Sep 2 '14 at 12:23
• Are those slides what you were asking about in your original question? Because I don't see the relationship between the question and those slides. The slides are showing a data cache, (not an instruction cache). In the slides the reads are not pipelined. They don't need to be because you can read the data and do the tag check simultaneously. If the tag check says "miss" then you just don't use the data. For a write you can't do the write simultaneously with the tag check because if it turns out that tag check says "miss" then you've just written invalid data to the cache line. – Wandering Logic Sep 2 '14 at 17:11

http://caps.cs.binghamton.edu/papers/aggarwal_ics_2005.pdf

I've found this paper which explains in the introduction that with higher associative caches the datapath to read data, compare tags and drive output becomes too long to use with high clock frequencies. Therefore the cache access is split into the 3 stages: decode set, compare tags and read, drive data. It goes further and explains that the access can have an extra stage added to avoid redundent reads of data and only read data after comparing and matching the correct tag, which improves power usage.

Although not really related to my question the paper also explains cache set and way prediction techniques to reduce the latency by parallelising address calculation and the cache read operation, the original paper refers to an article on the alpha 21264 microprocessor on how to implement the prediction. http://mixteco.utm.mx/~merg/AC/pdfs/alpha_21264.pdf

The technique is to reduce the latency by not waiting on the address calculation stages in the CPU pipeline. This is done by predicting the address of the data to be read and checking at the end of the operation.

• +1 for read address prediction. Like branch prediction, but for reading data. – gnasher729 Feb 27 '17 at 23:44

"If so then what microarchitectual changes can be made to allow CPU to pipeline operations so that it can reach 1 operation per clock assuming all hits."

It seems you are confusing latency here with throughput. Many CPUs have two read and one write port, and can read two items and write one item in the same cycle. So I assume you are asking about one cycle latency. And this is a really interesting question, and the answer is: You can make your CPU really slow.

L1 cache doesn't have multi-cycle latency just for fun. Addressing data in a large cache takes time, and more time the bigger the cache is. You need more time to select the correct data, you need more time because you have tons of connections to all the different cache lines. The only way to get the number of cycles latency down is (a) reduce the clock speed and (b) reduce the size of the cache. Both are things that you don't want.