Tensor Core
A tensor core is a small matrix-multiply-accumulate engine built into each GPU streaming multiprocessor: in one instruction it multiplies two low-precision matrix tiles and adds them into a higher-precision accumulator — the operation deep learning spends most of its time on.
One instruction does a whole matmul. A normal GPU (CUDA) core does one scalar multiply-add per cycle. A tensor core instead consumes two small matrix tiles — on the order of 16 x 16 — and produces D = A x B + C in a few cycles, so a single warp-level MMA instruction retires hundreds of multiply-accumulates. That is why a GPU's tensor-core FLOPS dwarf its general-purpose FLOPS.
Mixed precision is the trick. The multiplies run in a low-precision format (FP16, BF16, and on newer parts FP8 or FP4) while the running sum accumulates in FP32. Low precision makes the multipliers small and fast and cuts memory traffic; the wide accumulator keeps the sum from losing accuracy. You get the speed of low precision with much of the stability of high precision.
| GPU generation | Added formats | Notable |
|---|---|---|
| Volta (2017) | FP16 mul / FP32 acc | first tensor cores |
| Turing (2018) | INT8, INT4 | inference |
| Ampere (A100) | TF32, BF16 | structured sparsity |
| Hopper (H100) | FP8 | Transformer Engine |
| Blackwell (B200) | FP4 | 2nd-gen Transformer Engine |
Tensor core vs systolic array. Both win the same way — massive operand reuse inside a matrix engine — but the shapes differ. A TPU is one large systolic grid fed by the compiler; a tensor core is a compact MMA unit replicated across every SM and invoked as a GPU instruction over register-file tiles. Nvidia's are the best known, but AMD's CDNA matrix cores and Intel's XMX engines do the same job.
Read the tensor core through a quant lens rather than a marketing lens: a headline PFLOPS number only means something paired with its format (FP16 vs FP8 vs FP4) and whether it counts the 2:1 structured-sparsity mode. What decides real training and inference speed is achievable MACs per byte of HBM traffic at a tolerable precision — the tensor core raises the compute ceiling, but arithmetic intensity and accumulator width decide how much of it you actually reach.