GPU Tensor Core
GPU Tensor Cores are the specialized matrix multiplication accelerator units embedded in modern GPU architectures (NVIDIA Volta and later, AMD Matrix Cores) that perform small matrix multiply-accumulate operations (e.g., 4×4×4 or 16×16×16) in a single clock cycle at throughput rates 8-16x higher than standard floating-point units — enabling the massive FLOPS numbers (hundreds of TFLOPS) required for deep learning training and inference.
Architecture
Each Tensor Core performs a D = A × B + C operation on small matrix tiles in one cycle:
- Input matrices A, B: FP16, BF16, TF32, FP8, or INT8 depending on generation.
- Accumulator matrix C/D: FP32 or FP16 for higher precision accumulation.
- Throughput: NVIDIA H100 delivers 989 TFLOPS at FP16 Tensor and 1,979 TFLOPS at FP8. Standard FP32 ALUs deliver 67 TFLOPS — a 15-30x gap.
Hardware Generations
| Generation | GPU | Tile Size | Precisions | Peak TFLOPS |
|---|---|---|---|---|
| Volta (1st gen) | V100 | 4×4×4 | FP16→FP32 | 125 |
| Turing (2nd gen) | T4/RTX 2080 | 4×4×4 | FP16,INT8,INT4 | 130 |
| Ampere (3rd gen) | A100 | 8×4×8+ | FP16,BF16,TF32,FP64 | 312 |
| Hopper (4th gen) | H100 | 16×16×16 | FP16,BF16,FP8,INT8 | 989 |
| Blackwell (5th gen) | B200 | larger | FP4,FP6,FP8 | 4,500 |
Programming Model
- WMMA (Warp Matrix Multiply-Accumulate): CUDA API where a warp cooperatively loads matrix fragments from shared memory, performs the MMA operation, and stores the result. Each thread in the warp holds a portion of the matrix fragments.
- MMA PTX Instructions: Lower-level interface giving finer control over tile sizes and data layouts.
- cuBLAS/cuDNN: High-level libraries that automatically use Tensor Cores for GEMM and convolution operations. The recommended interface for most users — library kernels are highly tuned for each GPU generation.
Mixed Precision Training
Tensor Cores enable mixed-precision training: forward and backward passes compute in FP16/BF16 (fast Tensor Core operations), while master weights are maintained in FP32 for gradient accumulation accuracy. Loss scaling prevents gradient underflow in FP16. Result: 2-3x training speedup with negligible accuracy loss.
Feeding Tensor Cores
Tensor Cores can execute faster than data arrives from memory. Efficient utilization requires:
- Shared Memory Tiling: Load input matrices from global memory into shared memory tiles, then feed tiles to Tensor Cores. The software pipeline of load→compute→store must overlap to hide latency.
- Large Tile Sizes: Larger GEMM dimensions improve Tensor Core utilization. Small matrix operations (batch size 1 inference) under-utilize Tensor Cores and are better served by standard ALUs.
- Data Layout: Tensor Cores expect specific data layouts (column-major fragments). Memory access patterns must align with these requirements.
GPU Tensor Cores are the silicon embodiment of the observation that neural network computation is dominated by matrix multiplication — purpose-built hardware that delivers an order of magnitude more throughput for the single operation class that matters most for AI workloads.