Adaptive Inference

Adaptive Inference is the dynamic adjustment of a neural network's computational effort based on the difficulty of each input — allocating more computation to hard inputs and less to easy inputs, optimizing the average computation per sample while maintaining accuracy.

Adaptive Inference Mechanisms

• Early Exit: Skip later layers for confident predictions (BranchyNet, MSDNet).
• Dynamic Depth: Choose how many layers to execute per input (SkipNet, BlockDrop).
• Dynamic Width: Choose how many channels/filters to use per input (Slimmable Networks).
• Dynamic Resolution: Process easy inputs at lower resolution, hard inputs at higher resolution.

Why It Matters

• Efficiency: Easy inputs (majority in many applications) require much less computation — 2-10× average speedup.
• Budget-Aware: Set a computation budget and the network adapts to meet it.
• Semiconductor: Defect images vary in difficulty — simple good/bad decisions exit early, ambiguous defects get full computation.

Adaptive Inference is thinking harder when it matters — dynamically allocating computation based on each input's difficulty for efficient inference.