Attention Sink

Attention Sinks and StreamingLLM are the architectural phenomenon and inference technique where the first few tokens in a sequence consistently receive disproportionately high attention regardless of content — a pattern observed across virtually all Transformer models where initial tokens act as "attention sinks" that absorb excess attention mass, and the StreamingLLM method exploits this discovery to enable theoretically infinite context streaming by maintaining only the attention sink tokens plus a sliding window of recent tokens, providing constant-memory inference without quality degradation for indefinitely long conversations.

The Attention Sink Phenomenon

Observation: In virtually ALL transformers:
  Token 0 (BOS or first word) receives 20-50% of attention mass
  Token 1-3: Also receive elevated attention (5-15% each)
  Remaining tokens: Share the rest proportionally to relevance

Why?
  Softmax must sum to 1.0 across all tokens
  When no token is particularly relevant, attention mass must go SOMEWHERE
  First tokens become "default dump" for excess attention
  This happens REGARDLESS of the content of those tokens

Why Attention Sinks Exist

StreamingLLM

Problem: Standard sliding window attention fails catastrophically
  Window = tokens [101-200] (dropped tokens 0-100)
  Model expects attention sinks at positions 0-3 → they're gone → 
  Attention distribution collapses → quality tanks

StreamingLLM solution:
  Keep: [Token 0, 1, 2, 3] (attention sinks) + [last N tokens] (recent context)
  Drop: Everything in between
  
Example with window=4 sinks + 1000 recent:
  Context at step 5000: [0,1,2,3] + [4001,4002,...,5000]
  Context at step 50000: [0,1,2,3] + [49001,49002,...,50000]
  Memory: Always constant (1004 tokens)
  Quality: Comparable to full attention for recent-context tasks

Perplexity Comparison

Dedicated Attention Sink Token

# Training with a learnable sink token (prevents reliance on BOS)
class AttentionSinkModel(nn.Module):
    def __init__(self, base_model):
        super().__init__()
        self.model = base_model
        # Learnable sink token prepended to every sequence
        self.sink_token = nn.Parameter(torch.randn(1, 1, d_model))
    
    def forward(self, x):
        # Prepend sink token
        sink = self.sink_token.expand(x.size(0), -1, -1)
        x = torch.cat([sink, x], dim=1)
        return self.model(x)[:, 1:]  # remove sink from output

Implications for Model Design

• Models with explicit sink tokens: Better streaming performance.
• KV cache management: Always keep sink tokens, never evict them.
• PagedAttention: Pin sink token pages in memory.
• Positional encoding: Sink tokens should have fixed (not rotated) positions.

Applications of StreamingLLM

Limitations

• No recall of dropped tokens: Information between sinks and window is lost forever.
• Not a replacement for long context: Tasks requiring full document understanding still need full attention.
• Trade-off: Streaming capability vs. information retention.

Attention sinks and StreamingLLM are the key insight enabling infinite-length Transformer inference — by discovering that Transformers rely on initial tokens as attention reservoirs and preserving them alongside a sliding window, StreamingLLM provides constant-memory inference that runs indefinitely without quality collapse, solving a practical deployment problem for any application where conversations or data streams can grow without bound.