Adaptive Computation Time (Act)
Adaptive Computation Time (ACT) is a mechanism introduced by Alex Graves that allows recurrent neural networks and transformers to learn how many computational steps to perform for each input element, rather than using a fixed number of steps. ACT adds a learned "halting probability" at each step, and computation continues until the cumulative halting probability exceeds a threshold (typically 1.0), with a ponder cost penalty that encourages the model to halt early when additional computation is unnecessary.
Why ACT Matters in AI/ML:
ACT provides learned, input-dependent computation allocation that enables models to automatically dedicate more processing to difficult inputs and less to easy ones, achieving better quality-efficiency tradeoffs than fixed-depth architectures.
• Halting mechanism — At each computational step t, the model outputs a scalar halting probability h_t ∈ (0,1); the model halts at step N when the cumulative sum Σh_t first exceeds 1.0, with the remainder R_N = 1 - Σ_{t=1}^{N-1} h_t weighting the final step's contribution
• Ponder cost — An auxiliary loss term L_ponder = Σ R_t penalizes unnecessary computation, encouraging the model to halt as early as possible while still achieving accurate predictions; the ponder cost coefficient balances efficiency vs. accuracy
• Variable depth per position — In sequence models, each position can halt independently, meaning some tokens receive 2 computational steps while others receive 20, naturally allocating capacity where the input is most complex
• Differentiable computation budget — Unlike hard early-exit thresholds, ACT is fully differentiable through the halting mechanism, allowing end-to-end gradient-based training of both the computation and halting networks
• Universal Transformer integration — ACT combined with weight-sharing transformers (Universal Transformers) creates models that iterate transformer layers until convergence, with each position deciding independently when to stop
| Component | Function | Typical Value |
|---|---|---|
| Halting Unit | Outputs h_t per step | Sigmoid output, 0-1 |
| Cumulative Halt | Σh_t triggers stop | Threshold = 1.0 |
| Remainder | R_N = 1 - Σh_{1..N-1} | Weights final step |
| Ponder Cost | λ · mean(N+R) | λ = 10⁻² to 10⁻¹ |
| Max Steps | Hard upper limit | 10-50 steps |
| Avg Steps (easy) | Learned minimum | 2-5 steps |
| Avg Steps (hard) | Learned maximum | 10-30 steps |
Adaptive Computation Time is a foundational mechanism for input-dependent computation that enables neural networks to learn their own computational budget per input, automatically allocating more processing steps to difficult examples and fewer to easy ones, achieving superior efficiency-accuracy tradeoffs through end-to-end differentiable halting decisions.