I've got a few questions related to the different types of memory that we can find:

• (Good thing the pixel raster allows to discern a year of copyright - decades ago.) registers that are close physically to the CPU take another look: the registers are part of the CPU. RAM has a bigger memory size, the address in bits to access a register is longest should better read RAM has more cells, the address in bits to access one cell is longest to avoid naming confusion with CPU/*architectural* registers. (In the memory hierarchy, you seem to disregard bulk storage. Add non-volatile solid state memory, anyway.) Mar 17 '21 at 17:46
• – D.W.
Nov 3 '21 at 18:11

Why can't we simply put all of the RAM memory close physically to the CPU

Because that is not how geometry works. You cannot have everything close to the CPU, because the stuff that is close to the CPU is blocking the space close to the CPU, and thus other stuff has to be farther away.

Geometry works the same on the CPU as it does everywhere else: why don't all your neighbors live directly next door to you? Well, because your neighbors are taking up that space, so you simply can only have two direct neighbors. Everybody else has to live further away.

There are actually already 3-dimensional designs where the memory is stacked on top of the CPU, in order to be able to bring more memory closer to the CPU. But that only pushed the problem out, it does not solve it.

There is, however, a whole slew of other reasons, not just geometrical ones, why we have a memory hierarchy and not just a single flat memory. The main problems are that, in general, faster memory is also

• more expensive,
• more power hungry,
• more complex, and
• physically bigger (less dense).

That latter problem actually leads to an interesting optimization problem: as we said above, we can't have all the fast memory close to the CPU, because the space close to the CPU is limited. However, since slower memory is more dense, and thus takes up less physical space for the same memory capacity, you can actually put more slower memory close to the CPU than you can faster memory!

And even if you were to use only the fastest memory technology for all your memory, you quickly run into problems with the fundamental speed limit of the universe: the speed of light. As a general rule, our systems are designed in such a way that a single clock pulse must be able to propagate through the entire clock domain within a single clock cycle.

This means that no matter how close and how dense we pack our memory to the CPU, there comes a point where we cannot clock the memory with a high clock rate anymore. The further away we get, the lower the clock rate. But then it doesn't make sense to use the super-expensive, super-fast memory anymore, if we have to make it slower anyway. It makes more sense to use the cheaper, more dense, more energy-efficient memory.

And that is why the memory hierarchy exists: because the difference is not just speed, it is also cost, space, and energy. Here is an example of one possible memory hierarchy:

1. Registers (discrete flip-flops)
2. Level 0 Cache / µOP Cache (typically SRAM; per thread)
3. Level 1 Cache (typically SRAM; per core)
4. Level 2 Cache (typically SRAM; per core)
5. Level 3 Cache (typically SRAM; per die)
6. Level 4 Cache (typically SRAM; per module)
7. Local Main Memory (typically DRAM; per module)
8. Remote Main Memory (typically DRAM; "not this module")
9. SSDs (Flash; per node)
10. HDDs (magnetic; per node)
11. SAN (mix of SSDs and HDDs; per system)
12. Tape Library (per system)
13. Offsite Cloud Storage

Obviously, not all of these levels exist in all systems. This is an extreme example of a multi-core, multi-CPU, multi-node NUMA cluster with a separate storage network – my laptop doesn't have #6, #8, #10, #11, and #12, for example.

#8 only exists in so-called "NUMA" systems (Non-Uniform Memory Architecture), i.e. systems where not all RAM is "created equal". Typically, these are systems with a large number of CPUs, where it is not feasible to give all CPUs the same access to all of the RAM. Instead, the RAM is split into parts and typically one block of CPUs has direct access to one block of memory, and if it wants to access the rest of the memory, it typically has to talk to the CPU(s) that "own(s)" that block.

As a closing remark, I want to say that while the answers here have shown why your idea doesn't work, that does not mean that your idea is stupid! In fact, your idea is pretty much the Holy Grail of storage technology, and every couple of decades, a new technology comes along that tries to be fast, cheap, dense, energy-efficient, and ideally even non-volatile, at the same time, so that it can replace a large portion of that memory hierarchy.

However, as of today, none of those technologies have been able to live up to the hype, so to say: MRAM, FeRAM, T-RAM, Z-RAM, and many others. The latest entry being Intel's / Micron's 3D XPoint (better known under Intel's marketing brand Optane).

You can find a picture of an M1 chip where memory and CPUs are on the same physical chip. You’ll see that memory is bloody big, and you can’t move all of it close to the CPU.

The real reason is cost. L1 Cache can move data from/to several registers simultaneously and very fast. That makes it very expensive per byte. Nobody would be able to pay for even 1MB of L1 cache, and RAM is thousand times bigger. L2 cache uses a technology that is slower and cheaper but allows more storage, cheap enough that you can afford a few megabytes.

You will always have different technologies at different costs, and in that case you get the best results by mixing a tiny bit of super fast, super expensive, and a bit of slower, less expensive, and a lot of cheap and slow memory.

• I was talking theoretically honestly. So you're saying that we could have 16GB of L1 cache that would be faster than regular RAM ? Couldn't we just augment the size of the registers that are inside the CPU, or is that physically impossible (by augmenting. I mean having like 8GB at least)
– MM1
Mar 19 '21 at 8:48
• There is cheap, slow, big memory. And there is expensive, fast, tiny memory. And there is memory that is in between in price, speed and size. There is no technology that lets you create something like the memory that is currently used for L1 caches, at 16 GB size, for an amount of money that anybody would want to pay. Mar 19 '21 at 15:18