Early Exit
Early Exit and Conditional Computation are the inference efficiency techniques that allow neural networks to dynamically adjust the amount of computation per input — terminating processing at an intermediate layer when the model is already confident (early exit), or routing inputs through different subsets of the network based on difficulty (conditional computation), enabling 2-5x inference speedup on average while maintaining accuracy on the hard examples that need full computation.
Early Exit Architecture
Input → Block 1 → Classifier 1 → Confident? → YES → Output (fast!)
↓ NO
Block 2 → Classifier 2 → Confident? → YES → Output
↓ NO
Block 3 → Classifier 3 → Confident? → YES → Output
↓ NO
Block N → Final Classifier → Output (full computation)- Each intermediate classifier is a small head (linear layer) attached to intermediate features.
- Confidence threshold: If max softmax probability > τ → exit early.
- Easy inputs: Exit at block 1-2 (10-20% of computation).
- Hard inputs: Use all blocks (100% computation).
Benefits
| Metric | Without Early Exit | With Early Exit |
|---|---|---|
| Average latency | Same for all inputs | 2-5x faster on average |
| Easy input latency | Same as hard | 5-10x faster |
| Hard input accuracy | Baseline | Same (uses full model) |
| Average accuracy | Baseline | ≈ Baseline (threshold-dependent) |
Conditional Computation Approaches
| Approach | How | Example |
|---|---|---|
| Early Exit | Exit at intermediate layer | BranchyNet, DeeBERT |
| Mixture of Experts | Route to subset of experts | Switch Transformer, Mixtral |
| Token Dropping | Skip computation for uninformative tokens | Adaptive token dropping |
| Layer Skipping | Skip certain layers for easy inputs | LayerSkip, SkipDecode |
| Mixture of Depths | Route tokens to layers selectively | MoD (Mixture of Depths) |
Early Exit for Transformers (LLMs)
- DeeBERT: Attach classifier after each BERT layer → exit early for easy classification tasks.
- CALM (Confident Adaptive Language Modeling): Early exit for decoder LLMs.
- Each token can exit at different layer → some tokens need 4 layers, others need 32.
- Challenge: All tokens in a batch must reach the same layer → needs careful batching.
- LayerSkip (Meta, 2024): Train model with layer dropout → at inference, verify early exit with remaining layers → self-speculative decoding.
Mixture of Depths (MoD)
- Each transformer layer has a router that decides PER TOKEN whether to process it or skip.
- Top-k tokens (e.g., top 50%) routed through the full layer → others skip via residual connection.
- Result: 50% less compute per layer → model uses full depth for important tokens only.
Training Early Exit Models
- Joint training: Sum losses from all exit classifiers (weighted by layer depth).
- Self-distillation: Later exits teach earlier exits → improves early exit quality.
- Knowledge distillation: Full model (teacher) distills into early-exit model (student).
Practical Deployment
- Server-side: Vary computation based on query difficulty → reduce cost.
- Edge/mobile: Exit early to meet latency constraints → adapt to hardware.
- Cascading: Small model → medium model → large model (route by difficulty).
Early exit and conditional computation are essential techniques for cost-efficient AI deployment — by recognizing that not all inputs require the same processing depth, these methods allocate computation proportionally to difficulty, achieving significant speedups on average while preserving accuracy on the challenging cases that matter most.