Is loading from L2 or L3 of multi level cache direct or require a transfer to L1

Question

(This is mainly about reads, and commercial products not hypothetical machines.) When loading a CPU register with data contained in L2 or L3 of a multi-level cache. Can the data that is in L2 or L3 but not in L1 be read directly, or must the cache line be transferred into L1 before moving to a register?

Another way to ask; do L2 and L3 only act as cache for the next lower level (L3>L2, L2>L1) or do they have some direct use by the processor core, and can L2 be skipped (L3>L1)?

Most interested in the behavior of a modern multi-core with the amd64 ISA.(Though feel free to answer for IBM Power, ARM, or RISC-V if that is your specialty)

Is this use of L1(however it is used) only a matter of design policy or is it physically and logically predestined?

Answer 1

I am not aware of any x86 implementation that supported L1 cache bypassing for cacheable accesses unless one includes NT writes (which go to a special buffer and bypass all caches).

However, the MIPS R8000 (or specifically the R8010 FPU) had all FP memory accesses bypass L1, communicating directly with the L2 cache. For Itanium 2 (and follow-ons): "Floating-point loads are not cached in the L1D and are instead processed directly by the L2." (Intel® Itanium® 2 Processor Reference Manual For Software Development and Optimization, May 2004, p. 40)

Other implementations have had parallel L1 caches for which some accesses were allocated to a specific cache based on information in a page table entry.

The StrongARM SA1110 had an 8 KiB "main data cache" ("This cache allocates on loads to spaces marked B=1 and C=1.") and a 0.5 KiB "minicache" ("This cache allocates on loads to spaces marked B=0 and C=1"), where the B bit indicates "bufferable" and the C bit indicates "cacheable", being stored in a page table entry (Intel® StrongARM SA-1110 Microprocessor Developer’s Manual, June 2000).

The HP PA 7200 had an "assist cache" that was probed in parallel with the main L1 cache. All data accesses would be allocated to the assist cache; usually data evicted from the assist cache would be allocated to the main L1 cache. However, a "spatial locality only hint can be specified in load and store instructions ... Upon replacement, however, the line is flushed back to memory instead of being moved to the off-chip cache." ("Design of the HP PA 7200 CPU", Kenneth K. Chan et al., 1996) One could look at the main L1 cache as a huge direct-mapped victim cache for the assist cache, a reversing of size and associativity from the original victim cache proposal.

One interesting research example was the proposal for an Alpha processor with a vector extension ("Tarantula: A Vector Extension to the Alpha Architecture", Roger Espasa et al., 2002). Vector accesses would have bypassed the L1 data cache.

There is no insuperable barrier to implementing cache bypassing. There is additional wiring complexity (similar to higher associativity) and ideally one would want to know/accurately predict if a particular load is likely to be an L1 hit (out-of-order schedulers commonly assume L1 hits for scheduling dependent operations, if L1-bypassing was somewhat common such a design would have high replay overhead). With a writeback L1 cache, there is also some coherence difficulty if writes to the same cache block had been allocated to L1, but this could be handled by always probing L1 (but not reading data when bypassing is predicted) or marking in L2 that L1 has that data. Designing the L2 cache for somewhat frequent, sub-block accesses also has implications on area, power, and latency. Having a tiny cache in parallel with L1 would allow block-sized transfers that avoid polluting the main L1 and that cache might also be used as a L1 eviction buffer/victim cache/prefetch buffer.

Allocating non-temporal blocks to L1 with a replacement position/annotation for earlier eviction would have some of the benefit of complete bypassing with simpler timing factors, but even this is not (as far as I know) commonly implemented for L1 caches. (Intel has used set duelling to adjust replacement policies for L3 cache.)

Significant research has been done on cache bypassing.

Answer 2

Your processor may use read/write instructions to access hardware, in that case some pages will be marked as uncached, so a read of 4 bytes will directly read from an address that is most likely not connected to RAM.

Otherwise: I have never seen instructions that let you control this, the default is obviously going through L1 cache, so I assume there is no way to change this. It would be costly. If you read eight bytes from L2, and then access the same L2 cache line again, you will slow down. And it’s much easier if there is only one path from cache to registers.

The only slight adjustment that I have seen is that a cache line is divided into two halves, and the half containing the right data is transferred first to get the data a tiny bit quicker (but the other half is transferred immediately after that).

Note that you would have to check whether a value is in L1 cache anyway. And you have to build two transfer paths from L1 to registers and L2 to registers. Then if the value is changed you have to write it back to L2 cache, but at that point the cache line might already be evicted. For example x += 1 reads x, for some reason your thread is stopped and writing x+1 might happen a long time later. Now you have multiple cores with multiple threads using the same L2 cache and things get very complicated.