Universally Slimmable Networks

Universally Slimmable Networks (US-Nets) are an extension of slimmable networks that support any arbitrary width multiplier, not just preset values — enabling continuous, fine-grained accuracy-efficiency trade-offs at runtime.

US-Net Training

• Any Width: US-Nets support any width from the minimum to maximum (e.g., any value between 0.25× and 1.0×).
• Sandwich Rule: During training, always train the smallest and largest width (bread), plus $n$ random widths (filling).
• In-Place Distillation: The largest width acts as teacher — its soft labels guide the smaller widths.
• Switchable BN: Separate batch norm statistics for each width — essential for multi-width training.

Why It Matters

• Infinite Configs: Not limited to 4 preset widths — any width is available at runtime.
• Hardware Matching: Exactly match any hardware's computation budget — not just the nearest preset.
• Smooth Degradation: Performance degrades smoothly as width decreases — no sudden accuracy drops.

US-Nets are infinitely adjustable models — supporting any width configuration for perfectly fine-grained accuracy-efficiency control.