Batch Normalization Layer Normalization

Normalization Techniques in Deep Learning are the family of methods that standardize activations within neural networks to stabilize training dynamics, enable higher learning rates, and reduce sensitivity to weight initialization — with Batch Normalization, Layer Normalization, Group Normalization, and Instance Normalization each normalizing along different dimensions for different use cases.

Batch Normalization (BatchNorm):
- Operation: for each channel c, normalize activations across the batch dimension and spatial dimensions — μ_c and σ_c computed over (N, H, W) for each channel in a mini-batch; output = γ_c × (x - μ_c)/σ_c + β_c with learnable scale γ and shift β
- Training Behavior: running mean and variance computed via exponential moving average during training — stored statistics used during inference for deterministic behavior independent of batch composition
- Benefits: enables 10-30× higher learning rates, acts as regularizer (noise from mini-batch statistics), smooths the loss landscape — almost universally used in CNN architectures
- Limitations: performance degrades with small batch sizes (< 16) due to noisy statistics; not applicable to variable-length sequences; batch-dependent behavior complicates distributed training and inference

Layer Normalization (LayerNorm):
- Operation: normalizes across all features within each sample independently — μ and σ computed over (C, H, W) for each sample; no dependence on batch dimension
- Use Cases: standard in Transformer architectures (BERT, GPT, ViT) — batch-independent normalization essential for autoregressive models and variable-length sequence processing
- Pre-Norm vs. Post-Norm: Pre-LayerNorm (normalize before attention/FFN) provides more stable training for deep Transformers — Post-LayerNorm (original Transformer) requires learning rate warmup but may achieve better final accuracy
- RMSNorm: simplified variant using only root-mean-square normalization without centering — reduces computation by ~30% with comparable performance; used in LLaMA and other efficient Transformer architectures

Other Normalization Methods:
- Group Normalization: divides channels into G groups and normalizes within each group per sample — GroupNorm with G=32 achieves stable performance across all batch sizes; bridge between LayerNorm (G=1) and InstanceNorm (G=C)
- Instance Normalization: normalizes each channel of each sample independently over spatial dimensions — standard for style transfer where per-channel statistics encode style information that should be normalized away
- Weight Normalization: decouples weight vector magnitude from direction — reparameterizes W = g × v/ v with learned scalar g and unit direction v; more stable for RNNs than BatchNorm
- Spectral Normalization: constrains the spectral norm (largest singular value) of weight matrices — stabilizes GAN discriminator training by limiting the Lipschitz constant

Normalization techniques are among the most impactful innovations in deep learning practice — choosing the right normalization method for the architecture and use case directly determines training stability, convergence speed, and final model quality.