Rigging the Lottery

Rigging the Lottery (RigL) is a state-of-the-art Dynamic Sparse Training algorithm — that uses gradient information to intelligently regrow pruned connections, achieving dense-network-level accuracy while training with a fixed sparse computational budget.

What Is RigL?

- Key Innovation: Use the *gradient magnitude* of currently-zero (inactive) weights to decide which connections to grow back.
- Algorithm:
1. Drop: Remove $k$ active weights with smallest magnitude.
2. Grow: Activate $k$ inactive weights with largest gradient (gradient tells us "this connection *would* have been useful").
3. Maintain constant sparsity.
- Paper: Evci et al. (2020, Google Brain).

Why It Matters

• Performance: First sparse training method to match dense baselines on ImageNet at 90% sparsity.
• Efficiency: 3-5x training FLOPs savings vs dense training.
• Principled: The gradient-based grow criterion is theoretically motivated.

RigL is intelligent network rewiring — using gradient signals as a compass to navigate the space of sparse architectures during training.