Conditional Computation

Conditional Computation is the execution of only a subset of a neural network's parameters for each input — using gating mechanisms to selectively activate modules, layers, or experts, reducing the average computational cost while maintaining model capacity.

Conditional Computation Methods

• Gated Layers: Binary gates decide whether to execute each layer — skip unnecessary layers per input.
• Mixture of Experts: Route inputs to a subset of experts based on a learned gating function.
• SkipNet: Train a policy network to decide which residual blocks to skip for each input.
• Stochastic Depth: Randomly skip layers during training (regularization), deterministically during inference.

Why It Matters

• Decoupled Capacity and Compute: A model with 10B parameters can use only 1B per input — large capacity, small cost.
• Sparse Models: Conditional computation enables sparse, efficient models that scale beyond dense network limits.
• Switch Transformers: Google's Switch Transformer uses conditional computation to scale to trillion-parameter models.

Conditional Computation is activating only what's needed — selectively executing network components based on each input for massive efficiency gains.