Attention Rollout in ViT

Attention rollout in ViT is the layer-wise aggregation method that composes attention matrices across depth to estimate end-to-end token influence on final predictions - instead of viewing one layer in isolation, rollout traces how information propagates from input patches to output tokens.

What Is Attention Rollout?

• Definition: Recursive multiplication of attention matrices with identity residual terms across transformer layers.
• Core Idea: Influence accumulates through many blocks, so global attribution must include the full chain.
• Output: A single influence map showing patch contribution to CLS or target token.
• Scope: Works for classification and can be adapted to dense token outputs.

Why Attention Rollout Matters

• Deeper Explainability: Captures cross-layer pathways missed by single-layer heatmaps.
• Consistency Checks: Detects if influence remains stable across augmentations and seeds.
• Bias Detection: Highlights unintended dependencies on background regions.
• Model Comparison: Enables fair explainability comparison across ViT variants.
• Debugging Efficiency: Reduces manual review time by summarizing layer dynamics.

How Rollout Is Computed

Step 1:
- Collect attention matrices A_l from each layer and average or select heads.
- Add identity matrix to model residual mixing, then normalize rows.

Step 2:
- Multiply adjusted matrices from shallow to deep layers to obtain cumulative influence matrix.
- Extract influence from output token to input patch tokens.

Step 3:
- Reshape influence vector to patch grid and overlay as saliency map.
- Validate map behavior against counterfactual image edits.

Implementation Notes

• Head Aggregation: Mean aggregation is stable baseline, max can overemphasize outliers.
• Numerical Stability: Use float32 for matrix products in long depth models.
• Residual Handling: Identity blending choice strongly affects attribution sharpness.

Attention rollout in ViT is a robust way to summarize multi-layer information flow and patch influence in one interpretable map - it turns raw attention tensors into actionable explainability signals for model governance.