Gradient Flow Preservation

Gradient Flow Preservation is a design principle for pruning and sparse training — ensuring that removing weights does not disrupt the backpropagation signal, keeping gradient magnitudes stable across layers to prevent training collapse.

What Is Gradient Flow Preservation?

• Problem: Aggressive pruning can create "dead zones" where gradients vanish, causing layers to stop learning.
• Metrics: Checking the Jacobian singular values, layer-wise gradient norms, or signal propagation theory.
• Solutions:
• Balanced Pruning: Ensure each layer retains a minimum number of connections.
• Skip Connections: ResNet-style shortcut connections maintain gradient highways even if main path is heavily pruned.
• Dynamic Regrowth: DST methods (RigL) regrow connections in gradient-starved regions.

Why It Matters

• Trainability: A pruned network that can't propagate gradients is useless regardless of its theoretical capacity.
• Depth Sensitivity: Deeper networks are more fragile. Preserving flow is critical for 100+ layer architectures.

Gradient Flow Preservation is keeping the neural highway open — ensuring that information can flow backward for learning no matter how sparse the network becomes.