Dynamic Width Networks

Dynamic Width Networks are neural networks that adaptively select how many channels or neurons are active in each layer for each input — using fewer channels for simple inputs and more for complex ones, providing a continuous trade-off between accuracy and computation.

Dynamic Width Methods

• Slimmable Networks: Train a single network to operate at multiple preset widths (0.25×, 0.5×, 0.75×, 1.0×).
• Channel Gating: Learn binary gates to activate/deactivate channels per input.
• Width Multiplier: MobileNet-style uniform width scaling across all layers.
• Attention-Based: Use attention mechanisms to softly select channels.

Why It Matters

• Hardware-Friendly: Changing width maps directly to computation reduction on hardware (fewer MACs, less memory).
• Single Model: One trained model serves multiple width settings — no need to train separate models.
• Smooth Trade-Off: Width provides a smooth, continuous accuracy-efficiency trade-off.

Dynamic Width is adjusting the neural channel count — using more neurons for hard inputs and fewer for easy ones within a single flexible network.