Tensor Cores
Tensor Cores — specialized hardware units in NVIDIA GPUs that perform matrix-multiply-accumulate (MMA) operations at enormous throughput, designed to accelerate deep learning and HPC workloads.
What Tensor Cores Do
- Single operation: D = A × B + C, where A, B, C, D are small matrices (e.g., 4×4 or 8×4)
- One tensor core: Computes a 4×4×4 matrix multiply in a single cycle
- One SM has multiple tensor cores → massive parallel matrix throughput
Performance by Generation
| GPU Generation | Tensor Core | Peak (FP16) | Notes |
|---|---|---|---|
| V100 (Volta) | 1st gen | 125 TFLOPS | First tensor cores |
| A100 (Ampere) | 3rd gen | 312 TFLOPS | Added TF32, INT8, sparsity |
| H100 (Hopper) | 4th gen | 990 TFLOPS | Added FP8, transformer engine |
| B200 (Blackwell) | 5th gen | 2250 TFLOPS | 2x Hopper |
Supported Precisions
- FP16, BF16: Standard training precision
- TF32: 19-bit format, drop-in for FP32 matrix ops (Ampere+)
- FP8 (E4M3, E5M2): Hopper+ for inference
- INT8, INT4: Quantized inference
- FP64: For HPC scientific computing (A100+)
How to Use
- PyTorch: `torch.matmul()` with `torch.cuda.amp` (automatic mixed precision) → tensor cores used automatically
- Requires specific matrix dimension alignment (multiples of 8 or 16)
Tensor cores deliver 10-20x higher throughput than standard CUDA cores for matrix operations — they're why GPUs dominate AI training.