Context Length Extension

Context Length Extension is the set of techniques for enabling LLMs trained on short sequences to process much longer sequences at inference time — expanding usable context from 4K to 128K, 1M, or more tokens.

Why Context Length Matters

• 4K tokens ≈ 3,000 words ≈ 6 pages.
• 128K tokens ≈ 100,000 words ≈ entire novel.
• Long context enables: full codebase reasoning, book summarization, long document QA, multi-turn dialogue.

The Length Generalization Problem

• Models trained on 4K sequences struggle with 8K at inference — position IDs out-of-distribution.
• Attention scores become noisy at long ranges not seen during training.
• RoPE frequencies need adjustment for longer contexts.

Extension Techniques

RoPE Scaling:
- Linear Interpolation: Scale position indices by context_extension / train_length. Simple, loses some accuracy.
- NTK-Aware Scaling: Distributes interpolation across frequency dimensions — better quality.
- YaRN (Yet Another RoPE extensioN): Dynamic NTK + attention temperature scaling. Used in LLaMA 3 (128K).
- LongRoPE: Non-uniform RoPE rescaling per dimension — extends to 2M tokens.

Architecture Changes:
- Grouped-Query Attention (GQA): Fewer KV heads — reduces KV cache size linearly.
- Sliding Window Attention (Mistral): Each token attends to only W nearby tokens — O(NW) instead of O(N²).

Efficient Attention for Long Contexts:
- FlashAttention-2/3: Enables 100K+ context without OOM.
- Ring Attention: Distribute long sequences across multiple GPUs.

KV Cache Compression:
- SnapKV: Evict less-attended KV cache entries.
- StreamingLLM: Attend to initial tokens + recent window.
- H2O: Heavy-Hitter Oracle — keep most-attended keys.

Context length extension is a critical frontier in LLM capability — closing the gap between model context and real-world document lengths unlocks entirely new application categories.