A CPU is designed to make a single stream of instructions finish as fast as possible. It spends most of its transistors not on arithmetic but on hiding latency for one thread: large caches, deep out-of-order pipelines, branch predictors, speculative execution. It has a handful of these very sophisticated cores. This is the right design when the work is a sequence of unpredictable, dependent decisions — parse this file, follow this pointer, branch on this condition.
A GPU makes the opposite bet. It assumes the work is thousands of near-identical operations with no dependence on each other — the same arithmetic applied to a huge array of data. So it spends its transistors on arithmetic units, not on latency-hiding cleverness, and packs in thousands of small, simple cores. It hides latency not by predicting the future but by having so many threads ready to run that when one stalls waiting on memory, another instantly takes its place.
SIMT: threads in lockstep
The mechanism NVIDIA calls SIMT — Single Instruction, Multiple Thread. Threads are grouped into warps of 32, and all 32 threads in a warp execute the same instruction at the same time, each on its own data element. One instruction fetch drives 32 lanes of arithmetic. That's how a GPU gets its throughput: it amortizes the expensive control logic across 32 threads instead of paying for it per thread like a CPU.
The catch is right there in the design. If those 32 threads hit an if and take different branches — branch divergence — the hardware can't run both paths at once. It runs the then-branch with the else-threads masked off doing nothing, then runs the else-branch with the others masked off. Divergent, branchy code turns a 32-wide machine into a 1-wide machine. This is precisely why the pointer-chasing, decision-heavy code that a CPU eats for breakfast runs terribly on a GPU — and vice versa.
Arithmetic intensity: the real limiter
Whether a GPU's arithmetic units are fed depends on arithmetic intensity — the ratio of math operations to bytes moved from memory. Moving a byte from memory costs far more time and energy than doing a floating-point operation, so a kernel that does little math per byte is memory-bound: the arithmetic units sit idle waiting for data, and the GPU's thousands of cores are wasted. To actually use a GPU you need high arithmetic intensity — lots of FLOPs per byte loaded.
Vector add: c[i] = a[i] + b[i] loads 8 bytes, does 1 add → intensity ≈ 0.125 FLOP/byte → MEMORY-BOUND. Cores starve. Matrix multiply (N×N): moves ~N² data, does ~N³ FLOPs → intensity ∝ N → each element loaded is reused N times → COMPUTE-BOUND. Cores stay fed. This is what GPUs want.
Why matmul flies — and why AI runs on GPUs
Matrix multiplication is the ideal GPU workload on every axis. It is massively parallel — every output element is an independent dot product. It is branch-free — the same multiply-accumulate, no data-dependent ifs, so no warp divergence. And it has high, tunable arithmetic intensity — each input element gets reused across a whole row or column, so with the right tiling the arithmetic units stay saturated. A GPU built to run 32 identical operations in lockstep on data-parallel arrays is, almost by definition, a matrix-multiply engine.
And a neural network is, computationally, a stack of matrix multiplies. A transformer's attention and feed-forward layers are dense matmuls; training and inference are dominated by them. That is the whole reason the AI boom runs on GPUs and not CPUs: the fundamental operation of deep learning happens to be the exact shape of work GPUs were designed — originally for graphics — to devour. Modern accelerators lean into this so hard they add dedicated tensor cores that do nothing but small matrix multiplies. The hardware and the algorithm found each other.
Why it matters: "GPU vs CPU" isn't a speed contest, it's a question of matching work to machine. Latency-bound, branchy, pointer-chasing code belongs on a CPU; throughput-bound, regular, high-intensity array math belongs on a GPU. When someone says "just move it to the GPU and it'll be faster," the right question is: is this work actually GPU-shaped? For matrix multiply the answer is an emphatic yes — which is why the most valuable computation on earth right now is running on hardware originally built to draw triangles.