Training compute has grown at a staggering pace — Sevilla et al. found that since the early 2010s the compute used in landmark models has doubled roughly every six months, far outstripping Moore's law. That curve is why cost stopped being a rounding error and became the thing that decides who can train at the frontier. But "cost" is more than a headline dollar figure. Let's build it from the ground up.
The obvious part: GPUs × hours × utilization
Start with the compute. You need a certain number of floating-point operations to train a model; divide by your hardware's real throughput to get GPU-hours, then multiply by price. The trap is the word real. Nobody sustains a GPU's peak FLOPS — after communication stalls, memory limits, and pipeline bubbles, large runs often sit at 30–50% Model FLOPS Utilization. Your effective cost per useful FLOP is therefore two to three times the marketing number.
Rough training cost (compute only): cost ≈ (GPU-hours) × (price per GPU-hour) GPU-hours = total_FLOPs / (peak_FLOPs/s × MFU × 3600) Toy example — 2,000 GPUs, 30 days, $2/GPU-hr: 2000 × 30 × 24 × $2 = $2.88 M ← rental only At 40% MFU you paid for 100% of those hours but got 40% of the compute. Halve MFU and you double the real bill.
The parts nobody quotes
- Interconnect — at thousands of GPUs, the InfiniBand/NVLink fabric, switches, and cabling are a serious fraction of capital cost, and (per the last page) the all-reduce it carries is what caps your utilization. You pay for it twice: to buy it, and in the MFU it fails to save.
- Storage & data — datasets, checkpoints written every few minutes (terabytes each), and a parallel filesystem fast enough to feed thousands of GPUs without starving them.
- Power & cooling — a large cluster draws tens of megawatts continuously. Patterson et al., costing out models like GPT-3, showed energy and the resulting carbon are first-order, not a footnote — and every watt into the GPUs is roughly another watt of cooling. Over a multi-week run, the electricity bill alone rivals the hardware amortization.
- Failed runs — the line item nobody prices in. Hardware fails, loss diverges, a config bug surfaces at day 20. Real programs budget for restarts and abandoned runs; the compute you paid for that produced no model is pure cost.
Buy vs build, honestly
All of this feeds one decision: rent or own. Cloud GPU-hours carry a premium but zero commitment — right for bursty or one-off training. Owning hardware is cheaper per hour only if you keep it busy: a $30M cluster idle half the time costs double per useful hour, and now you own the megawatts, the cooling, the failed-run risk, and a depreciating asset in a market where next year's GPU is faster. The honest model isn't "cloud vs on-prem sticker price." It's total cost per useful FLOP at your real utilization, with power, people, and failed runs included. Do that arithmetic and the answer is usually less flattering to "just buy the GPUs" than the vendor slide suggests.
Why it matters: teams routinely under-budget training by ignoring utilization, power, and failed runs — then blow past the number by 2–3× and can't explain why. If you can build the full stack — GPU-hours ÷ real MFU, plus interconnect, storage, megawatts, and a restart budget — you can size a run before you spend, and make buy-vs-build a calculation instead of a vibe. That model is the single most useful spreadsheet in modern AI infrastructure.