The comforting assumption behind most software is fail-stop: a component either works correctly or halts. Memory follows this most of the time because of ECC — error-correcting codes add redundant bits so a single flipped bit is corrected on the fly and a double flip is at least detected and reported. The interconnect does the same with CRC checksums on every packet. This machinery is the reason you can trust a memory read at all.
Bit flips are not rare
How often does a bit actually flip? Schroeder, Pinheiro and Weber instrumented Google's entire server fleet for 2.5 years and found DRAM errors are far more common than lab studies had predicted — a large fraction of DIMMs saw correctable errors every year, at rates of thousands to tens of thousands of errors per DIMM per year on affected modules. Crucially, errors were dominated by hard (recurring) faults, not random cosmic-ray hits, and once a DIMM started erroring it tended to keep erroring. ECC caught the vast majority. But ECC has limits: it corrects one bit and detects two, so a triple-bit flip in the wrong place can slip through as a wrong value that looks perfectly valid.
The ones that don't announce themselves
ECC and CRC protect data at rest and in flight. They do nothing about data being computed. If a CPU or GPU executes a multiply and silently returns the wrong product, no checksum flags it — the corruption is born inside the arithmetic unit. This is silent data corruption (SDC), and two 2021 papers dragged it into the open:
- Google's "Cores that don't count" reported individual CPU cores — they call them mercurial cores — that intermittently miscompute specific operations, undetected by manufacturing tests, sometimes only under certain temperatures or voltages, sometimes worsening as silicon ages.
- Meta's "Silent Data Corruptions at Scale" described the same phenomenon across their fleet: a single defective core corrupting application data, taking months of engineering to root-cause because the symptom (a bad computation deep in a data pipeline) was so far removed from the cause (one instruction, on one core, occasionally wrong).
Both papers land on the same uncomfortable point: as feature sizes shrink and chips grow more complex, a nonzero fraction of cores in any large fleet will quietly produce wrong answers, and you will only find them if you go looking.
# end-to-end verification: don't trust the hardware, # check the result against something you can recompute. checksum_before = crc32(tensor) # producer send(tensor) assert crc32(tensor) == checksum_before # consumer verifies # fleet defense against mercurial cores: # run the SAME op on the SAME data on two cores, # compare — a mismatch means one core is lying.
Defense is end-to-end, not per-component
The lesson from decades of distributed-systems work is the end-to-end argument: a checksum on each hop doesn't guarantee correctness, because corruption can happen in the gaps between checks — in a register, in a bus, in an ALU. The only real guarantee is verifying the final result against something independently derived: application-level checksums that travel with the data, periodic re-execution of critical computations, and fleet-wide screening that runs known workloads and flags any core whose answer disagrees with the rest.
Why it matters: a frontier training run touches astronomical numbers of floating-point operations across tens of thousands of accelerators for weeks. A single silently-wrong gradient can nudge the model in a bad direction, and because it's silent you get no crash to tell you — just a run that mysteriously diverges or a loss curve that quietly gets worse, wasting millions of dollars of compute before anyone suspects the hardware. This is why hyperscalers now run continuous SDC-screening fleets, checkpoint aggressively (page 07) so a suspect segment can be replayed, and increasingly build verification into the training loop itself. "Trust the hardware" stopped being a safe assumption the moment we started running one computation across a whole datacenter.