Layer Normalization
Layer Normalization Pre-LN vs Post-LN Architecture determines where normalization occurs relative to residual connections in transformer blocks — Pre-LN (normalizing before sublayers) enabling training stability and better gradient flow for deep models while Post-LN (normalizing after additions) theoretically preserving more representational capacity.
Post-LN (Original Transformer) Architecture:
- Residual Block Structure: input x → sublayer (attention/FFN) → LayerNorm → output: (x + sublayer(x)) normalized
- Mathematical Form: y_i = LN(x_i + sublayer(x_i)) where LN(z) = (z - mean(z))/sqrt(var(z) + ε) — normalizes across feature dimension D
- Representational Capacity: post-normalization preserves original residual amplitude — sublayer outputs retain original scale before normalization
- Training Challenges: gradient magnitude inversely proportional to layer depth — deep networks (>24 layers) suffer vanishing gradients (0.1-0.01 gradient per layer)
- Stability Issues: post-LN requires careful initialization (small embedding scale 0.1, attention scale √d_k) — training becomes brittle with learning rate sensitivity
Pre-LN (Modern Architecture) Architecture:
- Residual Block Structure: input x → LayerNorm → sublayer (attention/FFN) → output: x + sublayer(LN(x))
- Mathematical Form: y_i = x_i + sublayer(LN(x_i)) — normalization applied before transformation
- Gradient Flow: residual connection carries constant gradient 1.0 throughout depth — enabling stable training of very deep models (100+ layers)
- Implicit Scaling: normalized inputs restrict to unit variance, naturally scaling sublayer outputs — reduces initialization sensitivity
- Easier Optimization: learning rate becomes less critical, wider range of hyperparameters work (LR 1e-4 to 1e-3) — robust training across model sizes
Technical Comparison:
- Residual Learning: post-LN preserves residual as original scale, pre-LN normalizes residual — mathematical difference with gradient implications
- Layer Skip Strength: post-LN enables stronger skip connections (amplitude 1.5-2.0x), pre-LN weaker (amplitude ~1.0x) — affects information flow
- Output Distribution: post-LN produces outputs with higher variance (std 1.5-2.0), pre-LN more constrained (std 1.0) — impacts downstream layer assumptions
- Initialization Dependency: post-LN requires embedding scaling 0.1-0.2, pre-LN works with standard 1.0 — critical for stable training
Empirical Performance Data:
- GPT-2 (Post-LN, 24 layers): requires LR 5e-5 with warmup schedule, trains unstably with LR 1e-3 — careful tuning needed
- GPT-3 (Post-LN, 96 layers): achieves 175B parameters despite depth, requires extensive grid search for hyperparameters
- Transformer-XL (Pre-LN): simplifies to relative position embeddings with pre-LN, trains stably without special initialization
- Llama 2 (Pre-LN): uses pre-LN throughout with RoPE, achieves 70B parameters with fewer training tricks — 20% fewer tokens needed for same performance
Practical Implications:
- Depth Scaling: pre-LN enables efficient scaling to 100+ layer models where post-LN becomes infeasible — key for retrieval-augmented and deep reasoning models
- Fine-tuning Stability: pre-LN allows larger learning rates (5e-5 to 1e-4) without divergence — beneficial for parameter-efficient fine-tuning
- Batch Size Sensitivity: post-LN training sensitive to batch size effects, pre-LN more robust — enables flexible batch sizing in distributed training
- Numerical Stability: pre-LN naturally keeps activations near normal distribution — reduces overflow/underflow in mixed precision training (FP16, BF16)
Recent Architecture Trends:
- RMSNorm Adoption: simplifying layer normalization to RMS(z) × γ without centering — 5-10% speedup with pre-LN, used in Llama and PaLM
- Parallel Attention-FFN: computing attention and FFN in parallel with pre-LN — enables faster training (1.5x throughput) in modern architectures
- ALiBi Integration: combining pre-LN with Attention with Linear Biases (ALiBi) — avoids positional embedding learnable parameters while maintaining efficiency
Layer Normalization Pre-LN vs Post-LN Architecture is fundamental to transformer design — Pre-LN enabling stable training of deep models and becoming standard in modern architectures like Llama, PaLM, and recent foundation models.