2
$\begingroup$

(This is mainly about reads, and commercial products not hypothetical machines.) When loading a CPU register with data contained in L2 or L3 of a multi-level cache. Can the data that is in L2 or L3 but not in L1 be read directly, or must the cache line be transferred into L1 before moving to a register?

Another way to ask; do L2 and L3 only act as cache for the next lower level (L3>L2, L2>L1) or do they have some direct use by the processor core, and can L2 be skipped (L3>L1)?

Most interested in the behavior of a modern multi-core with the amd64 ISA.(Though feel free to answer for IBM Power, ARM, or RISC-V if that is your specialty)

Is this use of L1(however it is used) only a matter of design policy or is it physically and logically predestined?

$\endgroup$
4
  • $\begingroup$ Related: en.wikipedia.org/wiki/Cache_inclusion_policy, cs.stackexchange.com/q/105949/755 $\endgroup$
    – D.W.
    Nov 20, 2023 at 6:22
  • $\begingroup$ @D.W. The difference between inclusion policies is not my question. I'm interested in loads to the core registers which in most cases are not full cache lines. (512b vector registers being the most common exception.) I may need to clarify my question later, no more time today though. $\endgroup$
    – Max Power
    Nov 20, 2023 at 6:43
  • $\begingroup$ See also en.wikipedia.org/wiki/CPU_cache#Exclusive_versus_inclusive. I'm not an expert on this subject, but it seems like that contains information that might help answer your question. In particular, it sounds like for inclusive architectures, the situation you mention cannot happen; for exclusive architectures, Wikipedia says data is transferred into L1; for non-exclusive architectures, these considerations don't dictate the answer. It sounds like this gives some example architectures as well as some search terms to research further. Or maybe I'm misunderstanding something. $\endgroup$
    – D.W.
    Nov 20, 2023 at 10:22
  • $\begingroup$ @D.W. In a cache exclusive architecture, when data is transferred from L2 to L1 that portion of the L2 cache is marked as unused, since there is no need to have two copies. If your caches are 128K L1 and 2048K L2 this gives you effectively 2176K of cache. $\endgroup$
    – gnasher729
    Nov 22, 2023 at 13:57

2 Answers 2

2
$\begingroup$

I am not aware of any x86 implementation that supported L1 cache bypassing for cacheable accesses unless one includes NT writes (which go to a special buffer and bypass all caches).

However, the MIPS R8000 (or specifically the R8010 FPU) had all FP memory accesses bypass L1, communicating directly with the L2 cache. For Itanium 2 (and follow-ons): "Floating-point loads are not cached in the L1D and are instead processed directly by the L2." (Intel® Itanium® 2 Processor Reference Manual For Software Development and Optimization, May 2004, p. 40)

Other implementations have had parallel L1 caches for which some accesses were allocated to a specific cache based on information in a page table entry.

The StrongARM SA1110 had an 8 KiB "main data cache" ("This cache allocates on loads to spaces marked B=1 and C=1.") and a 0.5 KiB "minicache" ("This cache allocates on loads to spaces marked B=0 and C=1"), where the B bit indicates "bufferable" and the C bit indicates "cacheable", being stored in a page table entry (Intel® StrongARM SA-1110 Microprocessor Developer’s Manual, June 2000).

The HP PA 7200 had an "assist cache" that was probed in parallel with the main L1 cache. All data accesses would be allocated to the assist cache; usually data evicted from the assist cache would be allocated to the main L1 cache. However, a "spatial locality only hint can be specified in load and store instructions ... Upon replacement, however, the line is flushed back to memory instead of being moved to the off-chip cache." ("Design of the HP PA 7200 CPU", Kenneth K. Chan et al., 1996) One could look at the main L1 cache as a huge direct-mapped victim cache for the assist cache, a reversing of size and associativity from the original victim cache proposal.

One interesting research example was the proposal for an Alpha processor with a vector extension ("Tarantula: A Vector Extension to the Alpha Architecture", Roger Espasa et al., 2002). Vector accesses would have bypassed the L1 data cache.

There is no insuperable barrier to implementing cache bypassing. There is additional wiring complexity (similar to higher associativity) and ideally one would want to know/accurately predict if a particular load is likely to be an L1 hit (out-of-order schedulers commonly assume L1 hits for scheduling dependent operations, if L1-bypassing was somewhat common such a design would have high replay overhead). With a writeback L1 cache, there is also some coherence difficulty if writes to the same cache block had been allocated to L1, but this could be handled by always probing L1 (but not reading data when bypassing is predicted) or marking in L2 that L1 has that data. Designing the L2 cache for somewhat frequent, sub-block accesses also has implications on area, power, and latency. Having a tiny cache in parallel with L1 would allow block-sized transfers that avoid polluting the main L1 and that cache might also be used as a L1 eviction buffer/victim cache/prefetch buffer.

Allocating non-temporal blocks to L1 with a replacement position/annotation for earlier eviction would have some of the benefit of complete bypassing with simpler timing factors, but even this is not (as far as I know) commonly implemented for L1 caches. (Intel has used set duelling to adjust replacement policies for L3 cache.)

Significant research has been done on cache bypassing.

$\endgroup$
1
$\begingroup$

Your processor may use read/write instructions to access hardware, in that case some pages will be marked as uncached, so a read of 4 bytes will directly read from an address that is most likely not connected to RAM.

Otherwise: I have never seen instructions that let you control this, the default is obviously going through L1 cache, so I assume there is no way to change this. It would be costly. If you read eight bytes from L2, and then access the same L2 cache line again, you will slow down. And it’s much easier if there is only one path from cache to registers.

The only slight adjustment that I have seen is that a cache line is divided into two halves, and the half containing the right data is transferred first to get the data a tiny bit quicker (but the other half is transferred immediately after that).

Note that you would have to check whether a value is in L1 cache anyway. And you have to build two transfer paths from L1 to registers and L2 to registers. Then if the value is changed you have to write it back to L2 cache, but at that point the cache line might already be evicted. For example x += 1 reads x, for some reason your thread is stopped and writing x+1 might happen a long time later. Now you have multiple cores with multiple threads using the same L2 cache and things get very complicated.

$\endgroup$
1
  • $\begingroup$ I know hardware cache cannot be directly controlled with instructions. However, the modes of operation of a cache can influence how a program can be best structured to take advantage of the cache, or at a minimum try to avoid the worst-case scenarios. In the last paragraph, This is why I was only talking about reads. but fair point on the write side I was thinking of a private(per core) L2 cache, and general purpose cache management logic would not know that my intent is reading from a block with no chance of a write to that block. 0x01+0x41 store 0x41; 0x01 to 0x40 being read only. $\endgroup$
    – Max Power
    Nov 28, 2023 at 4:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.