Copying Heads

Copying heads is the attention heads that facilitate direct or indirect copying of tokens from prior context into output prediction pathways - they are central to tasks that require exact string continuation and pattern reproduction.

What Is Copying heads?

• Definition: Heads route token identity information from source positions toward next-token logits.
• Use Cases: Important in code, lists, names, and repeated-structure generation.
• Mechanism: Often interacts with induction and residual stream composition components.
• Identification: Detected via token-tracing experiments and copying-specific prompt tests.

Why Copying heads Matters

• Behavior Insight: Explains exact-match continuation strengths in language models.
• Safety Relevance: Related to potential memorization and data leakage concerns.
• Performance: Copying pathways can improve fidelity on structured tasks.
• Failure Modes: Overactive copying can contribute to repetitive or context-locked outputs.
• Editing Potential: Targetable mechanism for controlling copy bias in generation.

How It Is Used in Practice

• Copy Benchmarks: Use prompts requiring exact token carryover to measure head contribution.
• Causal Ablation: Disable candidate heads and observe drop in exact-copy performance.
• Mitigation: Apply targeted interventions if copying creates undesirable memorization behavior.

Copying heads is a central mechanistic pattern for context-token reuse in transformers - copying heads provide a concrete bridge between attention dynamics and exact-sequence generation behavior.