Every program is a mix of work that can happen in parallel and work that must happen in sequence. Reading a config file, allocating the model, the final reduction that combines everyone's results — those are serial: one thing after another, no matter how many cores you own. Gene Amdahl, arguing in 1967 against the then-fashionable idea that you could reach any performance just by ganging up processors, wrote down the consequence. It's the single most important number in parallel computing, and most people never do the arithmetic.
The equation, and the number that stings
Let P be the fraction of your program that parallelizes and (1−P) the serial fraction that doesn't. With N processors, the parallel part runs in P/N of its original time, but the serial part is stuck at (1−P). So the speedup is:
1
S(N) = ─────────────────
(1 − P) + P / N
As N → ∞ : S(∞) = 1 / (1 − P)
Serial fraction 5% (P = 0.95):
S(∞) = 1 / 0.05 = 20× ← the ceiling
N = 10 → 6.9×
N = 100 → 16.8×
N = 1000 → 19.6× (you bought 1000, you got 20)
Sit with that. Ninety-five percent of your code is perfectly parallel, and you still can't beat 20×. Push the serial fraction to 10% and the ceiling drops to 10×. This is why "scale it horizontally" quietly stops working: past a point, you're paying for processors that spend their lives waiting on the one part of the program that runs alone.
Gustafson's rebuttal: change the question
In 1988, John Gustafson — then at Sandia, working on a 1024-processor machine — pointed out that Amdahl assumes a fixed problem size. In practice, people who buy 1024 processors don't run yesterday's problem faster; they run a bigger problem in the same time. If the serial part stays roughly constant while the parallel work grows with the machine, the serial fraction shrinks, and useful speedup scales nearly linearly. This is weak scaling (fixed work per processor) versus Amdahl's strong scaling (fixed total work, more processors). Both are true. Which one governs you depends entirely on whether your problem grows with your hardware.
Why this rules distributed training
Data-parallel training is the textbook case. Each GPU processes a slice of the batch — beautifully parallel — but every step ends with an all-reduce that sums gradients across all GPUs before the next step can begin. That synchronization is Amdahl's serial fraction wearing a modern costume. Add GPUs and the per-GPU compute shrinks while the all-reduce cost grows with the group; eventually communication dominates and throughput plateaus. That plateau is why serious training runs lean on weak scaling (bigger global batches, bigger models as you add hardware — the Gustafson move) and pour engineering into the interconnect: overlapping communication with compute, tree/ring reductions, fatter fabric. When a lab says its cluster "doesn't scale past 4,000 GPUs," they've hit Amdahl on the all-reduce, not a bug.
Why it matters: before you approve a bigger cluster, ask what fraction of the workload is genuinely serial — the synchronization barriers, the checkpoint writes, the metadata step. That fraction, not the vendor's peak FLOPS, sets your real ceiling. Amdahl tells you when more nodes stop helping; Gustafson tells you how to change the problem so they help again. Knowing which regime you're in is worth more than the next hardware order.