Input-Dependent Computation

Input-Dependent Computation is the paradigm where the computational graph or resource allocation of a neural network changes dynamically based on the input — the model "decides" how much and what type of computation to apply to each input, enabling efficient and flexible inference.

Forms of Input-Dependent Computation

• Routing: Mixture of Experts (MoE) — route each input to a subset of expert networks.
• Gating: Conditional computation gates decide which modules to activate per input.
• Attention: Self-attention dynamically weighs which features to focus on per input.
• Resolution: Choose input or feature map resolution based on input complexity.

Why It Matters

• Computational Efficiency: Not all inputs need the same computation — input-dependent allocation saves resources.
• Expressivity: The model can allocate specialized computation (different experts) for different input types.
• Scaling: MoE models scale to trillions of parameters while keeping per-input FLOPs constant.

Input-Dependent Computation is compute on demand — dynamically choosing what and how much to compute based on each individual input.