Saliency

Saliency Maps and Gradient Attribution are the earliest and most widely used class of explainability methods that identify which input regions most influenced a neural network's prediction — by computing gradients of the output with respect to input features to produce heatmaps highlighting the pixels, tokens, or features the model relied upon.

What Are Saliency Maps?

• Definition: Visualizations that assign an importance score to each input element (pixel for images, token for text) indicating how much that element influenced the model's prediction — produced by analyzing gradients from the output back to the input.
• Core Intuition: If slightly changing a pixel causes a large change in the predicted class probability, that pixel is "salient" — the model's prediction depends on it.
• Output: A heatmap overlaid on the input — warm colors (red/yellow) indicate high saliency, cool colors (blue) indicate low saliency.
• History: Vanilla gradient saliency (Simonyan et al., 2014) is one of the oldest deep learning explainability methods, predating LIME and SHAP.

Why Saliency Maps Matter

• Model Debugging: Identify when models use wrong features — a skin cancer classifier highlighting the ruler in dermatology images rather than the lesion reveals a dangerous spurious correlation.
• Trust Building: Show clinicians, radiologists, and domain experts what features drove the AI's decision — enabling validation of AI reasoning before clinical adoption.
• Bias Detection: Reveal whether models attend to protected attributes (face color in images, gender-coded words in text) when making predictions on sensitive tasks.
• Scientific Discovery: In scientific AI applications, saliency reveals which molecular features or genomic regions drive predictions — generating testable hypotheses.
• Regulatory Compliance: Provide required explanations for automated decisions in regulated domains (credit, healthcare, hiring).

Saliency Methods Taxonomy

Gradient-Based Methods:

Vanilla Gradient (Sensitivity Map):
- Compute ∂f(x)/∂x_i — gradient of predicted class score with respect to each input pixel/token.
- Fast (single backward pass); noisy and sensitive to input perturbations.
- Limitation: Saturated neurons have zero gradient even if very important.

Gradient × Input:
- Element-wise product of gradient and input value: (∂f/∂x_i) × x_i.
- Reduces noise; captures both direction and magnitude of feature importance.

Guided Backpropagation:
- Modified gradient: zero out negative gradients during backpropagation.
- Produces cleaner, visually appealing saliency maps.
- Critical flaw: Springenberg et al. (2017) showed guided backprop produces the same map regardless of model parameters — it is effectively an edge detector on the input, not an explanation of the model.

SmoothGrad:
- Average gradients over N noisy versions of the input: E[∂f(x + ε)/∂x] where ε ~ N(0, σ).
- Reduces gradient noise at the cost of N forward-backward passes (typically N=50).

GradCAM (Gradient-weighted Class Activation Mapping):
- Weight feature map channels by the global average gradient, then average and ReLU.
- Produces coarse but reliable class-discriminative visualizations at the final convolutional layer.
- Widely adopted in medical imaging for showing which image regions drove classification.

GradCAM++ / EigenCAM / Score-CAM:
- Variants improving GradCAM accuracy, multi-target support, or removing gradient requirements.

Integrated Gradients:
- Axiomatic method satisfying sensitivity and completeness axioms — the gold standard gradient attribution method (see entry 834).

Applications by Domain

Critical Limitations

Gradient Saturation:
- ReLU activations produce zero gradients for inputs beyond the saturation threshold — even highly important features may have zero gradient.
- Solution: Integrated Gradients paths through the saturation region.

Input Sensitivity vs. Model Explanation:
- Saliency shows local gradient — not global feature importance or counterfactual explanation.
- "This pixel is salient" means "changing this pixel slightly changes the output" — not "this pixel is the reason for the prediction."

Faithfulness:
- Guided backprop is demonstrably unfaithful — produces the same result regardless of model weights.
- All gradient methods should be validated with faithfulness tests (feature deletion, pixel flipping).

Adversarial Vulnerability:
- Saliency maps can be adversarially manipulated — models can be trained to produce arbitrary saliency maps for any input without changing prediction accuracy.

Saliency maps and gradient attribution are the essential first vocabulary of neural network explanation — despite their limitations, gradient-based methods provide fast, intuitive visualizations that have driven adoption of AI in medical imaging, scientific research, and safety-critical applications by giving human experts a starting point for validating model reasoning.