Last fall I went on a tour of the Blue Waters supercomputer at the University of Illinois. I asked whether anyone ever used the entire computer. I was told that it was always working on multiple projects. That made me wonder about the usefulness of supercomputers. Perhaps Blue Waters is unusual in that it has to be shared by industry and the university - I don't know. I assume there's some overhead in managing the processors and memory of a single supercomputer. Would it be more cost effective to build smaller computers? Can anyone help me to understand the value of supercomputers? Or is it that sometimes they are dedicated to single projects?

  • $\begingroup$ Can you explain how this is a computer science question? Afaik, most users of supercomputers are natural science and scientific computing folks. $\endgroup$ – Raphael Jun 7 '15 at 12:09
  • $\begingroup$ @Raphael: this is a question about resource allocation in the design of computer systems. The users of computer systems of any kind are rarely computer scientists. $\endgroup$ – Wandering Logic Jun 7 '15 at 12:30
  • 1
    $\begingroup$ Do you have any more detail on the usage of Blue Waters? For example, suppose that there's usually one project using 90% of the computer and another few mopping up the remaining 10%: in that case, it looks like the computer's about the right size. But if there are usually 10 projects each using 10%, that's a whole different kettle of fish. $\endgroup$ – David Richerby Jun 7 '15 at 16:35
  • 2
    $\begingroup$ Sounds like computer science to me. Computer architecture, cluster computing, grid computing, etc. All related and all computer science. $\endgroup$ – Dave Clarke Jun 7 '15 at 20:27

A typical job on Blue Waters is using about 10% of the machine and consumes a total of 75 node hours. Blue Waters has about 27500 nodes, so that means some of those "75 node hour" jobs are running in just a couple of minutes. That allows scientists to use the machine somewhat interactively. (You can see the moving averages here: http://xdmod.ncsa.illinois.edu/#tg_usage:group_by_Jobs_none)

Supercomputers are just large collections of smaller computers. The main reason we collect them together in one place is that we can share the cost most efficiently that way. You are trying to create a computer that can do a lot of work, and for which the total cost of ownership (the total cost of the computer, the power, and the maintenance), is minimized over the lifetime of the computer.

There are several factors involved in the total cost of ownership: The cost of the equipment is one. To minimize the cost of ownership you want the equipment to be doing useful work as large a percentage of the time as possible (ideally 100% of the time, realistically somewhat less, like 95% would be considered good), until the equipment burns out or becomes obsolete. In contrast, the computer in your laptop or your phone is probably actually in use less than 10% of the time you own it (you are asleep 33% of the time, you are eating and relaxing about half the time you are awake, and even when you are "using" the computer, the processor is idle most of the time.)

The second is the cost of power. There are several parts of this: the first is the cost of the power itself. Part of that cost is consumed in transporting the power from the power plant to the computer. Part of it is lost in the computer's "power supply" (which is just converting AC power into DC power). A larger AC->DC converter can usually be made more efficient. Additionally computers turn useful electric power into waste heat. So you also need to pay to remove the heat. Again, larger air conditioners can usually be made more efficient than multiple small air conditioners.

The third is the cost of maintenance. By putting together a bunch of computers and designing them so that when one goes down the rest keep running you can amortize the cost of maintenance staff over a much larger number of computer nodes than you could if the nodes were all different and placed in different buildings (or cities).

The details: Blue Waters has 288 cabinets. Each cabinet has 96 "nodes". Each node is a pretty normal high-end computer. Most nodes have 2 AMD Opeteron 6276 processors running at 2.3GHz, and 64GByte of DRAM. About 1/6 of the nodes instead have a single AMD Opteron 6276, an NVidia K20 GPU, and 38GByte of DRAM. If you wanted, you could buy something similar to a "node" for about \$3000 or \$4000 and put it in your living room to play video games. Blue Waters has about 27648 nodes. https://bluewaters.ncsa.illinois.edu/hardware-summary

Each node probably consumes a bit more than 500 Watts, and turns that power into heat. If you had a node in your living room to play video games it wouldn't be a particularly big deal. It would consume some electricity from the wall socket and generate about as much heat as a small personal space heater. In the winter that would be kind of nice and cozy. In the summer you'd have to run your air conditioner more frequently to keep your house comfortable. If you had it running full power all day every day, your electricity bill would go up considerably, perhaps double what you are consuming now.

But when you put 27648 of them together it consumes about 15 Megawatts, and generates a correspondingly large amount of heat. The true engineering marvel of Blue Waters, like any large data center, is the building itself. It's an enormous refrigerated box. The Blue Waters building is particularly interesting because it is a fantastically efficient. About 85% of the power going into the building is actually used to run the nodes. I believe I read somewhere (can't find it at the moment) only 15% is lost in power conversion and removing waste heat. That's a lot better than what you'd get from the 500Watt gaming computer in your living room. You'd probably need a 750Watt "power supply" and another couple hundred Watts to run the air conditioner.


Let's put it all together. By putting together thousands of smaller computers and spreading the usage among many many people, we keep those computers running most of the time, sharing the resources in a very efficient way. It costs a lot of money to give people computers that sit idle most of the time. The best way to save money on computation is to have people share the computers so the computers are busy most of the time.

Blue Waters is much more than just the computers inside it. It is specially designed to be as power efficient as possible. Part of that involves putting it near power plants to reduce power losses in power transmission lines. Here's a satelite picture of the part of Champaign IL containing Blue Waters to demonstrate: enter image description here


supercomputers are extremely important in modern research. they are not always used at total capacity depending on a supply/ demand/ management dynamic, and a continual upgrade/ replacement cycle. there are massive supercomputers used in the defense industry for weapons simulations (matching one of the early rationales/ impetuses for the invention of the computer in WWII, computing projectile trajectories). this use is not highly publicized. the modern weapon simulations are for nuclear weapons and are highly classified. the simulations allow new weapons designs to be "tested" accurately via computational simulations only. the US even rejects export of advanced computing technology to other countries eg China for this reason, because supercomputing has national security implications if another country can do effective supercomputing simulations.

there are many other uses. they can be used to simulate product design dynamics. for example the Tide company needed to figure out how to mix different ingredients in their laundry soap in an optimal way, and supercomputers were used to help compute the optimum mix.

most supercomputers involve running multiple different projects. they are used as shared resources and the management has strategies for choosing projects based on their overall load, research value, etc.

the basic value of supercomputers is that very large scale computations simply cannot be run on "smaller" computers with less overall CPU capacity. but in the last decade there has been a major shift toward building supercomputers with "commercial off the shelf" technology (aka COTS) which decreases their price and they still have very high performance.

wikipedia mentions basic uses of supercomputers, this is a partial list.

  • 1970s / Weather forecasting, aerodynamic research (Cray-1).[83]
  • 1980s / Probabilistic analysis,[84] radiation shielding modeling[85] (CDC Cyber).
  • 1990s / Brute force code breaking (EFF DES cracker).[86]
  • 2000s / 3D nuclear test simulations as a substitute for legal conduct Nuclear Non-Proliferation Treaty (ASCI Q).[87]
  • 2010s / Molecular Dynamics Simulation (Tianhe-1A)[88]
  • 1
    $\begingroup$ in recent times supercomputers have strong ties to working on big data and deep learning $\endgroup$ – vzn Jun 7 '15 at 16:02
  • 3
    $\begingroup$ This address what seems to be the main aspect of the question: what's the point of having a supercomputer of a particular size if it's always going to be running multiple projects simultaneously? What's the point of having a computer of capacity C if people only need capacity C/2 and it's presumably cheaper to build two smaller computers than one big one? $\endgroup$ – David Richerby Jun 7 '15 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.