Superposition Hypothesis

Superposition hypothesis is the proposal that neural networks represent many features in shared dimensions by overlapping them rather than allocating one dimension per feature - it explains how models can encode rich information with limited representational capacity.

What Is Superposition hypothesis?

• Definition: Features are packed into the same neurons or directions with partial interference.
• Motivation: Dense models face pressure to represent more concepts than available clean axes.
• Interpretability Impact: Explains prevalence of polysemantic units and mixed activations.
• Modeling: Analyzed through sparse coding and feature dictionary frameworks.

Why Superposition hypothesis Matters

• Theory Value: Provides coherent explanation for observed representation entanglement.
• Method Design: Guides development of feature extraction tools that untangle overlaps.
• Editing Safety: Highlights risk of naive neuron interventions causing unintended collateral changes.
• Scalability Insight: Suggests why larger models still exhibit mixed internal features.
• Research Direction: Motivates sparse feature spaces as interpretability targets.

How It Is Used in Practice

• Feature Extraction: Use sparse autoencoders to test whether mixed units decompose into cleaner features.
• Interference Analysis: Measure behavior overlap when candidate features co-activate.
• Model Comparison: Evaluate superposition patterns across scales and architectures.

Superposition hypothesis is a key theoretical lens for understanding compressed internal representations - superposition hypothesis is useful when paired with empirical decomposition and causal behavior testing.