This is the last page, and it's the one that ties the other nineteen together. Everything this guide has covered — the scheduler, the parallel filesystem, the interconnect, the GPUs, memory bandwidth, quantization, agents — is a potential bottleneck. On any real job exactly one of them is the constraint at any given moment, and it is rarely the one you'd guess. The whole discipline of making things fast reduces to a single habit: find the actual bottleneck before you optimize anything. Optimize the wrong thing and you spend a week making the fast part faster while the slow part sets your wall-clock unchanged.
Why intuition is usually wrong
Programmers assume the bottleneck is compute because compute is what they wrote. But a modern node can do arithmetic far faster than it can feed itself data. The real limit is usually elsewhere:
- I/O. Reading a dataset off disk, or writing checkpoints, while the expensive GPUs sit idle waiting.
- Network. An all-reduce across nodes that stalls every rank until the slowest link finishes.
- A serial section. One un-parallelized stretch that Amdahl's law pins your whole speedup against — 5% serial code caps you at 20× no matter how many cores you add.
- Memory bandwidth. The GPU has the FLOPs but can't load operands fast enough; the cores wait on DRAM. This is the dominant limit in LLM inference.
- The scheduler. Your job spends more time queued than running (see page 03).
- The metadata server. Thousands of tiny file opens crushing the filesystem's metadata path (see page 04), while bandwidth sits unused.
Leiserson et al., in their 2020 Science survey of where post-Moore performance comes from, make the structural version of this point: with the transistor free-lunch over, the gains now live at the "Top" of the stack — software, algorithms, and how well work maps to hardware — not the "Bottom." Which is to say: the bottleneck moved into your code and your data movement. You have to go find it.
Measure first — the numbers overrule the story
The cardinal rule is that a measurement beats an argument every time. Before changing a line, profile: use perf to sample where CPU time goes, iostat and dstat to watch disk and network, and a timeline profiler like NVIDIA's Nsight Systems (nsys) to see CPU, GPU, CUDA calls, and memory traffic on one timeline. The timeline is where the truth lives — you can literally see the GPU sitting idle while a data-loader thread does all the work.
# Where is a GPU job actually spending time? Capture a timeline: $ nsys profile -o run python train.py # A common, damning result: GPU compute (kernels) ...... 22% <-- what you tried to speed up Idle, waiting on input ...... 61% <-- the REAL bottleneck Host<->Device copies ...... 17% # Conclusion: don't touch the kernel. Fix the data pipeline — # prefetch, more loader workers, keep the GPU fed.
Notice the arithmetic: even if you made those kernels infinitely fast, you'd cut only 22% of the wall-clock. Fix the 61% idle and you nearly triple throughput. That ratio — how much of the total does this part even represent — is the first question to ask of any optimization, and profiling is the only way to answer it honestly.
Why it matters: this is the through-line of the whole guide, and it's what separates people who run big machines from people who merely have access to them. A GPU cluster is a chain of subsystems — scheduler, storage, network, memory, compute — and the chain runs at the speed of its slowest link. The engineer who wins is not the one who knows the most about GPUs; it's the one who measures the whole job, finds the one link that's actually binding, and fixes that. Learn to profile before you optimize, and you'll never again waste a week speeding up the fast part.