Content-Based Sparse Attention

Content-Based Sparse Attention is a dynamic sparse attention mechanism where the sparsity pattern is determined by the input content — using hashing, clustering, or learned routing to identify which key-value pairs are most relevant to each query, attending only to those.

Key Approaches

• Reformer (LSH): Locality-Sensitive Hashing groups similar queries and keys into the same bucket.
• Routing Transformer: Learned routing assigns tokens to clusters, attention within clusters only.
• Clustered Attention: K-means clustering of queries/keys, attention within clusters.
• Top-$k$ Selection: Compute approximate attention scores, attend only to top-$k$ keys.

Why It Matters

• Adaptive: The sparsity pattern adapts to the input — important dependencies are never missed.
• Better Than Fixed: Can capture irregular, content-dependent long-range dependencies that fixed patterns miss.
• Challenge: The routing/hashing overhead must be small enough to justify the attention savings.

Content-Based Sparse Attention is attention that finds its own shortcuts — dynamically discovering which tokens matter most to each query.