Supermasks

Supermasks are a binary mask applied to a randomly initialized neural network that achieves good performance without any weight training — demonstrating that a sufficiently overparameterized random network already contains useful sub-networks.

What Is a Supermask?

• Concept: Instead of learning weights, learn which weights to keep (binary mask optimization).
• Process: Fix weights at random init $\theta_0$. Optimize mask $m \in {0,1}^n$. Inference: $m odot \theta_0$.
• Finding: A random dense network + learned mask can achieve ~95% of trained network accuracy on MNIST.

Why It Matters

• Extreme Efficiency: Only 1 bit per parameter (on/off) needs to be learned, not 32-bit floats.
• Theory: Supports the "Strong Lottery Ticket" hypothesis — that random networks contain solutions without training.
• Hardware: Could enable ultra-low-power inference with fixed random weights and binary masks.

Supermasks are finding intelligence in randomness — proving that the structure of connections matters more than the values of the weights.