Interpretability

Interpretability enables understanding of why models make specific predictions or decisions. Motivation: Trust, debugging, compliance (right to explanation), scientific understanding, safety verification. Approaches: Feature attribution: Which inputs influenced output (attention, gradients, SHAP, LIME). Mechanistic interpretability: Understand internal computations (circuits, neurons, features). Concept-based: Map representations to human-understandable concepts. Probing: What information is encoded in hidden layers. Post-hoc vs intrinsic: Explaining existing models vs designing interpretable architectures. For transformers: Attention visualization, layer-wise relevance propagation, probing classifiers, circuit analysis. Challenges: Faithfulness (explanations may not reflect actual reasoning), complexity of modern models, scalability. Tools: TransformerLens, Captum, Ecco, inseq. Applications: Understanding model failures, detecting spurious correlations, safety cases, model editing. Trade-offs: Interpretable models may sacrifice performance, post-hoc methods have faithfulness issues. Current state: Active research area, partial solutions exist, full mechanistic understanding distant. Critical for AI safety and trust.