Mechanistic

Mechanistic Interpretability is the branch of AI safety and interpretability research that reverse-engineers neural networks by identifying the specific algorithms, circuits, and features that implement model behaviors — pursuing complete, faithful understanding of how transformers compute rather than post-hoc approximations or correlational probes.

What Is Mechanistic Interpretability?

• Definition: The systematic effort to identify and understand the actual computational mechanisms inside neural networks — the specific neurons, attention heads, circuits, and features that causally produce observed model behaviors.
• Analogy: Mechanistic interpretability is to neural networks what neuroscience is to brains — or more precisely, what reverse engineering is to compiled software. The goal is to reconstruct human-readable pseudocode from the network's weights.
• Key Methods: Feature visualization, attention pattern analysis, activation patching (causal tracing), probing, sparse autoencoders, and circuit analysis.
• Primary Focus: Transformer language models — particularly GPT-2, GPT-4, and Claude — where the architecture is well-understood and the stakes of understanding alignment are highest.

Why Mechanistic Interpretability Matters

• AI Safety: If we can understand what computations a model performs, we can verify whether it has learned deceptive behaviors, dangerous knowledge, or misaligned goals — rather than hoping alignment training worked correctly.
• Debugging: Identify why models fail on specific inputs by tracing the computation that produced the failure — enabling targeted fixes rather than blind retraining.
• Alignment Verification: Confirm that safety training actually removed harmful behaviors rather than merely suppressing them — mechanistic verification vs. behavioral testing.
• Scientific Understanding: Build a true scientific theory of how neural networks learn and represent knowledge — foundational for the field of AI.
• Capability Prediction: Understand what new behaviors emerge from scale before deploying larger models.

Core Concepts

Features:
- The basic units of representation — directions in activation space that correspond to specific concepts.
- A "banana feature" fires when the model processes banana-related text.
- Features may be monosemantic (one concept per neuron) or polysemantic (multiple concepts per neuron — superposition).

Circuits:
- Subgraphs of the network (specific neurons + attention heads + weights) that implement a specific algorithm.
- Discovered by following information flow from input to output for specific behaviors.
- Example: The "indirect object identification" circuit in GPT-2 that identifies the indirect object in sentences like "John gave Mary the book."

Attention Heads:
- Transformers consist of attention heads that route information between token positions.
- Heads have identifiable functions: some copy information, some attend to previous similar tokens (induction heads), some identify syntactic structure.

Key Discoveries in Mechanistic Interpretability

Induction Heads (Anthropic, 2022):
- Specific attention head pairs that implement in-context learning — searching for the pattern [A][B]...[A] and predicting [B].
- Formed during a sudden phase transition in training, coinciding with the emergence of in-context learning ability.
- Suggests in-context learning has a specific, identifiable mechanical implementation in transformer weights.

Indirect Object Identification Circuit (Redwood Research / Anthropic):
- Analyzed GPT-2's circuit for completing "John gave Mary the book; Mary gave [John]."
- Identified 26 specific attention heads with specific roles: subject inhibition, name mover, backup name mover, S-inhibition, induction.
- Complete causal account of a specific linguistic capability.

Grokking and Modular Arithmetic:
- Models trained on modular arithmetic exhibit "grokking" — sudden generalization after overfitting.
- Mechanistic analysis revealed the model learned a specific Fourier frequency algorithm for modular arithmetic.

Superposition and Sparse Autoencoders:
- Models represent more features than dimensions by encoding features as nearly-orthogonal directions that overlap.
- Sparse autoencoders decompose these overlapping representations into interpretable monosemantic features.

The Circuits Approach

Step 1 — Identify a behavior: "The model correctly identifies the indirect object in double-object constructions."

Step 2 — Activation Patching: Systematically corrupt then restore activations at different network components to identify which are causally necessary.

Step 3 — Component Attribution: Determine which attention heads, MLPs, and residual connections contribute most to the behavior.

Step 4 — Weight Inspection: Directly inspect what those components compute from their weight matrices.

Step 5 — Reverse Engineering: Formalize the discovered algorithm in pseudocode and verify it generalizes.

Research Organizations

• Anthropic Interpretability Team: Core circuit analysis work; Claude interpretability; sparse autoencoder research.
• Redwood Research: IOI circuit; causal scrubbing for faithfulness testing.
• EleutherAI: Open-source models + interpretability tooling (TransformerLens).
• Chris Olah (Google Brain / Anthropic): Pioneered feature visualization and circuit discovery.

Mechanistic interpretability is the scientific program to make AI systems as understandable as the circuits inside a computer — as researchers scale circuit analysis from toy models to frontier AI, mechanistic interpretability promises to transform AI alignment from a behavioral art into an engineering discipline with formal verification of safety-critical properties.