Dynamic Depth Networks

Dynamic Depth Networks are neural networks that adaptively choose how many layers to execute for each input — skipping unnecessary layers for easy inputs to save computation, while using the full depth for challenging inputs that require more processing.

Dynamic Depth Mechanisms

• Early Exit: Attach classifiers at intermediate layers — exit when confident (BranchyNet, MSDNet).
• SkipNet: Learn a binary gate per residual block — decide to execute or skip each block.
• BlockDrop: Train a policy to select which blocks to execute, targeting a computation budget.
• Layer Dropping: Stochastically drop layers during training (regularization), prune at inference.

Why It Matters

• Computation Savings: Skipping 30-50% of layers saves proportional computation with <1% accuracy loss.
• Latency Prediction: The number of executed layers directly determines inference latency.
• Heterogeneous Deploy: The same model can run at different depths for different hardware budgets.

Dynamic Depth is thinking only as deep as needed — adaptively choosing the number of processing layers based on each input's complexity.