FP32
FP32 (Single-Precision Floating Point) is the 32-bit numerical format that serves as the baseline precision for neural network training — using 1 sign bit, 8 exponent bits, and 23 mantissa bits to represent numbers with ~7 decimal digits of precision across a range of ±3.4×10³⁸, providing the numerical stability needed for gradient computation and weight updates while consuming 4 bytes per parameter, making a 7B parameter model require 28 GB of memory in FP32 representation.
What Is FP32?
- Definition: The IEEE 754 single-precision floating-point format — 32 bits total with 1 sign bit (positive/negative), 8 exponent bits (range: 2⁻¹²⁶ to 2¹²⁷), and 23 mantissa bits (~7 decimal digits of precision). The standard numerical format for scientific computing and the default training precision for neural networks.
- Training Baseline: FP32 is the "gold standard" precision for training — all gradients, weights, activations, and optimizer states are computed and stored in FP32 by default, providing sufficient precision for the small gradient updates that drive learning.
- Memory Cost: 4 bytes per value — a 7B parameter model requires 28 GB just for weights in FP32, plus 2-3× more for optimizer states (Adam stores momentum and variance in FP32), making FP32 training memory-intensive.
- Master Weights: In mixed-precision training, a FP32 copy of all weights is maintained as "master weights" — FP16/BF16 is used for forward/backward computation, but weight updates are applied to the FP32 master copy to prevent precision loss from accumulating small gradient updates.
FP32 vs. Other Precisions
| Format | Bits | Exponent | Mantissa | Range | Precision | Memory/Param |
|---|---|---|---|---|---|---|
| FP32 | 32 | 8 | 23 | ±3.4×10³⁸ | ~7 digits | 4 bytes |
| TF32 | 19 | 8 | 10 | ±3.4×10³⁸ | ~3 digits | 4 bytes (internal) |
| BF16 | 16 | 8 | 7 | ±3.4×10³⁸ | ~2 digits | 2 bytes |
| FP16 | 16 | 5 | 10 | ±65504 | ~3 digits | 2 bytes |
| INT8 | 8 | N/A | N/A | -128 to 127 | Integer | 1 byte |
| INT4 | 4 | N/A | N/A | -8 to 7 | Integer | 0.5 bytes |
FP32 in the ML Workflow
- Training: FP32 master weights + FP16/BF16 compute = mixed-precision training — 2× speedup on tensor cores with minimal accuracy loss. FP32 accumulation prevents precision loss in reductions.
- Inference: FP32 is rarely used for inference in production — models are quantized to FP16, INT8, or INT4 for 2-8× memory reduction and faster execution.
- TF32 (Tensor Float 32): NVIDIA A100/H100 tensor cores transparently compute FP32 operations at TF32 precision (8-bit exponent, 10-bit mantissa) — providing 8× speedup over true FP32 with same range but reduced precision, enabled by default.
FP32 is the numerical foundation of neural network training — providing the precision and range needed for stable gradient computation and weight updates, while mixed-precision techniques and inference quantization reduce its memory and compute costs by using lower-precision formats where full FP32 accuracy is not required.