People picture a supercomputer as one enormous, exotic brain. It isn't. Walk the floor of El Capitan or Frontier and you're looking at rack after rack of what are, individually, unremarkable servers — a couple of CPUs, some GPUs, memory, a network card. What makes it a supercomputer is that all of those boxes are stitched together so a single job can spread across all of them at once. The current #1 machine, El Capitan at Lawrence Livermore, reaches 1.742 exaflops — 1.742 × 1018 floating-point operations per second — not from one fast chip but from roughly 11 million cores cooperating.
One program, ten thousand copies
The dominant model is embarrassingly literal: you launch the same program on every node, and each copy works on its own slice of the problem. This is SPMD — single program, multiple data. Split a weather simulation of the atmosphere into a grid, hand each node a chunk of the sky, and let them all step forward in time together. The catch is that the chunks aren't independent. Your patch of atmosphere is bordered by your neighbor's patch, so at every timestep you must exchange edge values with the nodes holding the adjacent cells. Multiply that across thousands of nodes and the real work of the machine becomes communication, not arithmetic.
rank 0 rank 1 rank 2 rank 3 [ A ] [ B ] [ C ] [ D ] | <-> | <-> | <-> | exchange halo cells v v v v step step step step then a barrier: nobody ---- ---- ---- ---- advances until all arrive
Why agreement is the hard part
The instant a program spans more than one machine, you inherit every hard problem in distributed systems — and you can't opt out of any of them. The nodes have no shared clock and no shared memory; the only way one node knows anything about another is by sending a message across the network. So the machine has to constantly answer questions that are trivial on your laptop: Are we all on the same timestep? Did everyone's data arrive? One node is running 3% slower than the rest — do the other 9,999 wait for it? That last one, load imbalance, is a silent killer: a job runs only as fast as its slowest participant, so a single straggler node throttles the entire machine.
Then there's correctness. At 10,000 nodes running for days, hardware failure isn't an edge case — it's the expected operating condition. A memory bit flips, a network link drops a packet, a node dies mid-computation. A correct supercomputing program has to produce the same answer whether it ran on 8 nodes or 8,000, and survive parts of itself failing along the way. This is why "make it agree" — consensus, synchronization, fault tolerance — is the discipline, and raw FLOPS are almost a footnote.
The Berkeley framing
Berkeley's influential "Landscape" reports made this the headline: the shift to parallelism wasn't a performance tweak, it was a change in what programming is. Their argument — aimed at the whole industry as clock speeds stalled and core counts exploded — was that the bottleneck would be our ability to express and coordinate parallel work correctly, not the availability of transistors. Fifteen years on, that's exactly the wall the AI world hit: the hardware is abundant, the coordination is the problem.
Why it matters: training a large model is the same problem wearing new clothes. You copy the model across thousands of GPUs, each processes a different slice of the batch, and then — every single step — they must agree on the updated weights via an all-reduce across the network. The GPUs are fast; the run is bounded by how quickly ten thousand of them can synchronize. Understand supercomputing as a coordination problem and modern AI infrastructure stops looking novel and starts looking familiar. Everything that follows in this guide is a piece of how the agreement gets done.