Flash Attention
FlashAttention is a memory-efficient, IO-aware exact attention algorithm that reduces GPU memory usage from O(N²) to O(N) and speeds up attention computation by 2-4x — enabling training of long-context LLMs without approximation.
The Standard Attention Problem
- Standard attention materializes the full N×N attention matrix (N = sequence length).
- For N=4096, this is 4096² × 2 bytes = 32 MB per head.
- For 32 heads × 40 layers = 41 GB just for attention matrices (bottleneck for long contexts).
- High-bandwidth memory (HBM) is much slower than SRAM: unnecessary memory reads/writes dominate runtime.
FlashAttention Key Insight
- Tiling: Split Q, K, V into blocks that fit in SRAM (fast on-chip memory).
- Online Softmax: Compute softmax incrementally — no need to store the full attention matrix.
- Fused Kernel: Single CUDA kernel performs Q×K, softmax, and ×V in one pass.
- Result: Never materialize the full N×N matrix — only store the O(N) output.
Performance Gains
| Metric | Standard Attn | FlashAttention | FlashAttention-2 |
|---|---|---|---|
| Memory | O(N²) | O(N) | O(N) |
| Speedup vs baseline | 1x | 2-4x | 4-8x |
| Max sequence (A100) | ~8K | ~64K | ~128K |
FlashAttention-2 Improvements
- Better thread block partitioning for modern GPUs.
- Parallelism across sequence dimension (not just batch/heads).
- ~2x faster than FlashAttention-1.
FlashAttention-3 (2024)
- Designed for Hopper (H100) GPUs.
- Exploits asynchronous execution and FP8 precision.
- ~1.5-2x faster than FlashAttention-2 on H100.
Adoption: PyTorch 2.0+ includes `torch.nn.functional.scaled_dot_product_attention` with FlashAttention built in.
FlashAttention is the critical engineering breakthrough that made 100K+ context LLMs practical — it enabled GPT-4's 128K context and Gemini's 1M context windows.