# Are GPU's only a fraction of the cost of supercomputers?

According to wikipedia, the Frontier supercomputer is the largest supercomputer in the world, at 1 exaFLOPS, and cost 600 million USD.

I am quite surprised that the largest supercomputer only has 1 exaFLOPs, because of the following:

• The AMD Instinct MI250X has about 100 teraFLOPS, i.e. 0.1 petaFLOPS.

• Hence, if you stick about 10000 MI250X's together, you get 1 exaFLOPS.

• The price of an AMD Instinct MI250X is about 10k USD, so 10000 of them are about 100 million USD.

• This would suggest that the actual workhorse of the supercomputer is only 20% of its cost (since Frontier cost 600 million USD).

Maybe I did my calculation incorrectly, but if not then this surprises me. Why are the GPU's such a small fraction of the actual cost?

ps. Not sure which stackexchange to use for this if computer science is the wrong one.

• Can you replace "largest" by "fastest"? Largest is by physical volume/area while fastest is by computing speed. Here is the largest computer ever built. Jun 18, 2022 at 16:45

You have to remember that sticking a bunch of GPUs together does not linearly increase the performance. You can argue that yeah, if all the GPUs are running then instructions are being run, but to actually efficiently run useful operations on a bunch of GPUs is no easy task.

Some factors for scaling up GPUs:

• Communication: this is a big one. If you have a bunch of GPUs, how do you broadcast instructions to all of them, and put them all back together?
• On this topic, you would need bridges or some sort of other hardware to communicate data. If you have to transfer large amounts of data across distances, you will lose time while you wait for data transfers.
• Benchmark for instructions: There's a bunch of literature (and scandals) about how chip companies purposefully devise very specific tests to boost their flop count in the disgust of their competitors. A big historical one has been CISC vs RISC processors.
• Architecture: Again, this has to do with linearity, but a supercomputer will have a very different design than your personal computer. A supercomputer will have specialized architecture and hardware just to make that 1 TF work, which will add to the cost. Also having a 1 of 1 will make the production cost more expensive than a mass produced item like an AMD graphics card.
• i think that's the wrong shot. while performance in real applications usually don't grow linearly, there are lot of them, and published SC performance figures are computed in a simple way. BTW, the same applies to the single GPU - its official perfromance is just peak performance computed by multiplying frequency by number of ALUs Jun 18, 2022 at 22:36

Labor cost. 6x is an amazingly great work!

DoD kept awarding modular cellular designs on infinitely scalable FPGA arrays for many decades, and those projects have failed inevitably due to the lack of a distributed parallel model of computation. We just bunch whole bunch of von-Neumann cores to each other, than try to comprehend and scale that mess.

Building a new supercomputer is a manual architecture every single time.

Writing software for every new supercomputer is a manual labor every single time.

Nobody still has solved infinite scalability problem.