Centered Kernel Alignment

Centered kernel alignment is the representation similarity metric that compares centered kernel matrices to quantify alignment between activation spaces - it is widely used for robust layer-to-layer and model-to-model representation comparison.

What Is Centered kernel alignment?

• Definition: CKA measures normalized similarity between two feature sets via kernel-based statistics.
• Properties: Invariant to isotropic scaling and orthogonal transformations in common settings.
• Usage: Applied to compare layer evolution, transfer learning effects, and training dynamics.
• Variants: Linear and nonlinear kernels provide different sensitivity profiles.

Why Centered kernel alignment Matters

• Robust Comparison: Provides stable similarity scores across models with different widths.
• Training Insight: Tracks representation drift during fine-tuning and continued pretraining.
• Architecture Study: Useful for identifying where two models converge or diverge internally.
• Efficiency: Computationally tractable for many practical interpretability studies.
• Interpretation Limit: High CKA does not guarantee identical functional circuits.

How It Is Used in Practice

• Layer Grid: Compute CKA across full layer pairs to identify correspondence structure.
• Data Consistency: Use identical stimulus sets and preprocessing for fair comparison.
• Cross-Metric Check: Validate conclusions with complementary similarity and causal analyses.

Centered kernel alignment is a standard quantitative tool for representation alignment analysis - centered kernel alignment is strongest when used as part of a broader functional-comparison toolkit.