Gradient-Based Pruning

Gradient-Based Pruning is a more principled pruning criterion — using gradient information (or second-order derivatives) to estimate the impact of removing a weight on the loss function, rather than relying on magnitude alone.

What Is Gradient-Based Pruning?

• Idea: A weight is important if removing it causes a large increase in loss.
• First-Order (Taylor): Importance $approx |w cdot partial L / partial w|$ (weight times gradient).
• Second-Order (OBS/OBD): Uses the Hessian to estimate the curvature of the loss landscape around each weight.
• Fisher Information: Uses the Fisher matrix as an approximation to the Hessian.

Why It Matters

• Accuracy: Can identify important small weights that magnitude pruning would incorrectly remove.
• Layer Sensitivity: Naturally adapts pruning ratios per layer based on gradient flow.
• Cost: More expensive than magnitude pruning (requires backward pass), but more precise.

Gradient-Based Pruning is informed surgery — using diagnostic information about the network's health to decide what to remove.