Each node on a cluster has its own memory. There is no shared array everyone can read. So when a job spans 4,000 nodes, the only way they cooperate is by explicitly sending data to each other — message passing. The standard for this, since the mid-1990s, is MPI, the Message Passing Interface: a specification (not a product) that defines how independent processes name each other, send and receive buffers, and — crucially — perform group operations together.
Ranks, and the two kinds of message
An MPI program launches the same executable as many ranks — say 32,000 copies, each with a unique integer ID from 0 to 31,999. Every rank runs the same code but branches on its rank number, so rank 0 might read the input file and hand pieces to the rest. Communication comes in two flavours:
- Point-to-point — one rank sends a buffer to one other rank.
MPI_Send/MPI_Recv. This is the raw primitive. - Collective — every rank in a group participates in one coordinated operation. This is where the real work is, and where the performance lives.
Collectives matter because the naive way to do a group operation — have everyone send to rank 0, let it compute, then send back — makes rank 0 a bottleneck that gets worse as you add nodes. The whole art is doing these operations so their cost grows slowly, ideally like the logarithm of the number of ranks, not linearly.
The collectives you actually need to know
- Broadcast — one rank has a value; every rank needs a copy. Done as a tree, it takes ⌈log₂ P⌉ steps, not P.
- Reduce — every rank has a number; combine them all (sum, max, min) down to one result on one rank.
- All-reduce — reduce, but the combined result lands on every rank. This is broadcast-plus-reduce, and it is the one that matters most.
- Gather / Scatter / All-to-all — collect pieces to one rank, hand pieces out, or have everyone exchange everyone-to-everyone.
How these are implemented is not obvious. Thakur, Rabenseifner and Gropp showed that the best algorithm depends on message size: for a short all-reduce you want a latency-optimal recursive-doubling scheme; for a long one you switch to a bandwidth-optimal reduce-scatter followed by all-gather, which moves far less data over the wire. Same MPI call, completely different machinery underneath, chosen at runtime.
// every rank contributes its local gradient sum;
// every rank gets back the global sum, in place.
MPI_Allreduce(MPI_IN_PLACE, grad, n,
MPI_FLOAT, MPI_SUM, MPI_COMM_WORLD);
// divide by world size → the averaged gradient
Why all-reduce is the heart of AI training
Data-parallel training puts a copy of the model on every GPU and feeds each a different slice of the batch. Each GPU computes gradients on its slice — but to take one coherent optimizer step, every GPU must apply the average gradient across all of them. That average is exactly an all-reduce with a sum, divided by the number of workers. It happens every single training step, on tensors that can be gigabytes.
Because those tensors live in GPU memory and the links are NVLink and InfiniBand rather than a switched Ethernet cluster, NVIDIA built NCCL (and AMD its API-compatible RCCL) — GPU-native collective libraries that speak the same broadcast/reduce/all-reduce vocabulary as MPI but are tuned for GPU topology. NCCL's default all-reduce is a ring: a scatter-reduce phase then an all-gather phase, sized so each GPU sends the same volume regardless of ring length. Every distributed PyTorch and JAX job you have ever run leans on this. The forty-year-old idea of a collective did not get replaced by AI — it got a GPU back-end and became load-bearing infrastructure for the whole field.
Why it matters: when a training run stalls, "the GPUs are slow" is usually wrong — the GPUs are waiting on an all-reduce. Communication, not computation, is what caps the scaling of large models, which is why overlapping all-reduce with backprop, picking the right collective algorithm, and getting the network topology right are among the highest-leverage things an infrastructure engineer can do. The MPI collective is the abstraction the entire industry is quietly optimizing around.