Winning Tickets

Winning Tickets are the specific sparse sub-networks identified by the Lottery Ticket Hypothesis — sub-networks that, when trained from their original random initialization, achieve comparable performance to the full dense network.

What Are Winning Tickets?

• Definition: A mask $m$ over weights $\theta_0$ such that training $m \odot \theta_0$ achieves accuracy $\geq$ training $\theta_0$ in $\leq$ iterations.
• Properties:
• Initialization Dependent: The ticket only works with its *original* random init, not a new random init.
• Transferable: Tickets found on one task often transfer to related tasks.
• Stable: Late Rewinding (resetting to iteration $k$ instead of $0$) improves stability for large networks.

Why They Matter

• Sparse Training: If we can identify tickets early, we can train only the essential connections from the start.
• Generalization: Winning tickets often generalize better (fewer parameters = less overfitting).
• Hardware: Could enable training directly on edge devices if tickets are found cheaply.

Winning Tickets are the diamonds in the rough — proving that neural network training is really a search problem for the right sparse structure.