Landmark Attention

Landmark attention is the attention strategy that introduces selected anchor tokens or summary landmarks to help models access long-range information efficiently - it reduces full quadratic attention cost while preserving global context access paths.

What Is Landmark attention?

• Definition: Sparse attention design where regular tokens attend through designated landmark nodes.
• Mechanism: Landmark tokens act as compressed hubs for long-range information routing.
• Complexity Benefit: Cuts attention compute relative to dense all-to-all attention.
• Long-Context Role: Supports longer sequences by improving memory and compute scalability.

Why Landmark attention Matters

• Efficiency: Enables longer inputs under fixed hardware budgets.
• Global Access: Maintains pathways for distant dependency handling.
• RAG Relevance: Useful when prompts include many retrieved chunks and long histories.
• Architectural Flexibility: Can be combined with other sparse or hierarchical attention methods.
• Tradeoff Management: Requires careful landmark design to avoid information bottlenecks.

How It Is Used in Practice

• Landmark Selection: Choose anchors by structure boundaries, salience scores, or learned policies.
• Hybrid Attention: Blend local dense windows with landmark-mediated global connections.
• Task Benchmarks: Evaluate long-range reasoning, factuality, and latency before deployment.

Landmark attention is an efficient long-context attention pattern for scalable transformers - well-chosen landmarks preserve global reasoning while reducing computational burden.