Pages 10–12 all pointed the same way: modern accelerators are fast because they move and multiply low-precision numbers in bulk. This page is about spending those bits wisely. "Mixed precision" means using a narrow format for the expensive bulk math and a wider one for the few operations that can't tolerate the error — getting most of the speed with none of the divergence.
The formats, and what each trades
A floating-point number splits its bits into exponent (dynamic range — how large or small a value it can represent) and mantissa (precision — how many significant digits). Every format is a different bet on that split:
format bits exp/mant the trade FP32 32 8 / 23 baseline; range and precision to spare TF32 19* 8 / 10 FP32's range, less precision; NVIDIA tensor-core default FP16 16 5 / 10 precise but NARROW range → gradients underflow to 0 BF16 16 8 / 7 FP32's range, coarse precision → no loss scaling needed FP8-E4M3 8 4 / 3 more mantissa; for weights/activations (forward) FP8-E5M2 8 5 / 2 more range; for gradients (backward) FP4 4 ~2 / 1 frontier; needs per-block scaling to stay usable *TF32 keeps 19 bits internally in a 32-bit container.
The key insight, from Micikevicius et al.'s 2018 "Mixed Precision Training," is that FP16's problem is range, not precision: gradient magnitudes in deep networks are often so small they round to zero in FP16's narrow exponent, and the training stalls. BF16 sidesteps this entirely by keeping FP32's 8-bit exponent — same range, coarser mantissa — which is why BF16 became the default for large-model training: it usually needs no special handling at all.
Two tricks that keep FP16 honest
When you do use FP16, two techniques from the original paper prevent the model from breaking:
- FP32 master weights. Keep a full-precision copy of the weights. Do the forward and backward pass in FP16 for speed, but apply the tiny weight updates to the FP32 master — otherwise small updates vanish into FP16 rounding and learning stops.
- Loss scaling. Before backprop, multiply the loss by a large constant. This shifts the whole gradient distribution up into FP16's representable range so small gradients survive; you divide it back out before the optimizer step. Without it, the left tail of the gradient histogram underflows to zero and the network never converges.
A third rule underlies both: keep sensitive accumulations wide. The tensor cores multiply in FP16/FP8 but accumulate the running sum in FP32, because summing thousands of small products in the narrow format loses too much. Softmax, layer-norm statistics, and the optimizer state stay in higher precision for the same reason. Low precision is for the bulk multiply; the reductions stay honest.
FP8 — and why current GPUs are "fast"
Micikevicius and colleagues' 2022 "FP8 Formats for Deep Learning" defined the two 8-bit encodings now baked into hardware: E4M3 (more mantissa, for the forward-pass weights and activations) and E5M2 (more range, for backward-pass gradients). They showed FP8 can match 16-bit training quality across CNNs, RNNs, and transformers up to 175B parameters — using each format where its range/precision trade fits. This is not an abstraction: the headline TFLOPS on a current data-center GPU (page 10) is an FP8 number, and it is roughly double the FP16 rate. The reason today's accelerators are "fast" is that they compute in 8 bits — and, on the newest generation, 4 — with per-block scaling standing in for the loss-scaling trick above. Mixed precision is not an optimization anymore; it is the substrate the whole AI-compute economy runs on.
Why it matters: every doubling of throughput from FP32 → FP16/BF16 → FP8 → FP4 came from removing bits — and each step reintroduced the same failure mode: silent divergence when a sensitive operation loses its range or its accumulation width. Knowing which ops can go narrow (the bulk matmuls) and which must stay wide (accumulations, norms, master weights) is the difference between a 2× speedup and a model that quietly stops learning. It's also why "what precision?" is now the first question about any training run's cost and stability.