Sparse Model Topology Updates

Dynamic Sparse Training (DST) is a training paradigm where the sparse network topology changes during training — allowing connections to be pruned and regrown dynamically, so the network can discover the optimal sparse structure while training.

What Is DST?

- Key Difference from Pruning: Pruning starts dense and removes. DST starts sparse and rearranges.
- Algorithm (SET/RigL):
1. Initialize a sparse random network.
2. Train for $Delta T$ steps.
3. Drop: Remove connections with smallest magnitude.
4. Grow: Add new connections with largest gradient.
5. Repeat.
- Budget: Total number of non-zero weights stays constant throughout.

Why It Matters

• Training Efficiency: Never allocates memory for dense matrices. The FLOPs budget is always sparse.
• Performance: RigL matches dense training accuracy at 90% sparsity.
• Exploration: Allows the network to explore different topologies and find better sparse structures.

Dynamic Sparse Training is neural plasticity — mimicking the brain's ability to rewire connections based on experience.