Previous Token Heads
Previous token heads is the attention heads that strongly attend to the immediately preceding token position - they provide local context routing that supports many higher-level circuits.
What Is Previous token heads?
- Definition: Attention pattern is concentrated on token index minus one relative position.
- Functional Use: Creates short-range context features used by downstream heads.
- Circuit Role: Often upstream of induction and local-grammar processing mechanisms.
- Detection: Identified through average attention maps and positional preference metrics.
Why Previous token heads Matters
- Foundational Routing: Local token transfer is a building block for many model computations.
- Interpretability Baseline: Simple positional behavior provides clear mechanistic anchors.
- Composition Insight: Helps explain how later heads build complex behavior from local signals.
- Error Analysis: Weak or noisy local routing can degrade syntax and continuation quality.
- Comparative Study: Useful for scaling analyses across model sizes and architectures.
How It Is Used in Practice
- Positional Probes: Measure head attention by relative position across diverse prompts.
- Circuit Mapping: Trace which later components consume previous-token features.
- Intervention: Ablate candidate heads and monitor local dependency performance drops.
Previous token heads is a basic but important positional mechanism in transformer attention - previous token heads are critical primitives for constructing higher-order sequence-processing circuits.