Duplicate Token Heads

Duplicate token heads is the attention heads that preferentially attend to earlier occurrences of the current token identity - they support repetition-aware processing and pattern tracking in context.

What Is Duplicate token heads?

• Definition: Heads locate prior same-token positions rather than purely positional neighbors.
• Behavior Role: Can help detect repetition structure and anchor continuation choices.
• Circuit Interaction: Often contributes to induction-like and copying-related pathways.
• Measurement: Identified by attention enrichment toward prior matching-token indices.

Why Duplicate token heads Matters

• Pattern Memory: Facilitates reuse of earlier sequence structure.
• Mechanistic Clarity: Demonstrates identity-based lookup behavior in attention.
• Failure Insight: May contribute to repetitive loops in generation if overactive.
• Tool Benchmark: Useful target for evaluating feature and circuit discovery methods.
• Scaling Analysis: Helps compare emergence of token-matching behavior across checkpoints.

How It Is Used in Practice

• Controlled Prompts: Use synthetic repetition prompts to isolate duplicate-token behavior.
• Causal Testing: Patch or ablate candidate heads and quantify repetition-handling changes.
• Interaction Study: Map dependencies between duplicate-token heads and induction heads.

Duplicate token heads is an interpretable identity-matching motif in attention systems - duplicate token heads highlight how transformers use token-identity lookup to support sequence-level behavior.