Dynamic Neural Networks

Dynamic Neural Networks are neural networks whose architecture, parameters, or computational graph change during inference — adapting their structure based on the input, resource constraints, or other runtime conditions, in contrast to static networks with fixed computation.

Types of Dynamic Networks

• Dynamic Depth: Vary the number of layers executed per input (early exit, skip connections).
• Dynamic Width: Vary the number of channels or neurons per layer (slimmable networks).
• Dynamic Routing: Route inputs through different paths in the network (MoE, capsule routing).
• Dynamic Parameters: Generate parameters conditioned on the input (hypernetworks, dynamic convolutions).

Why It Matters

• Efficiency: Adapt computation to input difficulty — easy inputs use less computation.
• Flexibility: One model serves multiple deployment scenarios with different resource budgets.
• State-of-Art: Large language models (GPT-4, Mixtral) use dynamic routing (MoE) for efficient scaling.

Dynamic Neural Networks are shape-shifting models — adapting their own architecture and computation at inference time for maximum flexibility and efficiency.