Dynamic Architecture
Dynamic Architecture refers to neural networks that change their computational structure — topology, depth, width, or connectivity — at runtime based on the properties of the input data, creating input-specific computation graphs rather than applying a fixed architecture uniformly to all inputs — a paradigm shift from static neural networks where every input traverses the same computational path regardless of its complexity, structure, or information content.
What Is Dynamic Architecture?
- Definition: Dynamic architecture encompasses any neural network design where the computation graph is not fixed at model definition time but is determined (partially or fully) during inference based on the input. This includes conditional execution (skip layers), structural adaptation (build graph to match input structure), and resource-adaptive computation (adjust width/depth based on compute budget).
- Static vs. Dynamic: A standard CNN or transformer is static — the same sequence of operations (convolutions, attention layers, feed-forward blocks) is applied to every input regardless of content. A dynamic architecture applies different operations, different numbers of operations, or different connectivity patterns depending on what the input requires.
- Historical Context: Dynamic computation has deep roots — recursive neural networks (TreeRNNs) that build structure matching parse trees, graph neural networks that process arbitrary graph topologies, and hypernetworks that generate task-specific weights have all explored aspects of dynamic architecture. Modern dynamic architectures unify these ideas with learned routing and conditional computation in transformer-scale models.
Why Dynamic Architecture Matters
- Information-Proportional Compute: Static networks waste computation on "easy" regions of input data. A face detection CNN processes sky pixels with the same compute as face pixels. Dynamic architectures allocate more computation to information-dense regions and less to uniform or predictable regions, improving the compute-per-quality ratio.
- Structural Alignment: Some data types have inherent structure that static architectures cannot exploit. Tree-LSTMs match their network topology to the syntactic parse tree of a sentence. Graph neural networks match their message-passing topology to the molecular graph. Dynamic architectures align computation with data structure rather than forcing data through a fixed pipeline.
- Scalability: Dynamic architectures enable scaling model capacity (total parameters) without proportionally scaling inference cost. Mixture-of-Experts models store 8x the parameters of an equivalent dense model but activate only 1/8 per token. This decouples capacity from cost, enabling much larger models within fixed compute budgets.
- Multi-Modal Fusion: Dynamic architectures naturally handle multi-modal inputs (text + image + audio) where different modalities require different processing pathways. A dynamic router can send text tokens through language layers, image patches through vision layers, and route cross-modal tokens through fusion layers — all within a single model.
Dynamic Architecture Examples
| Architecture | What Varies | Mechanism |
|---|---|---|
| MoE (Mixture of Experts) | Width — which expert processes each token | Gating network routes tokens to top-k experts |
| MoD (Mixture of Depths) | Depth — how many layers each token traverses | Per-layer router decides execute or skip |
| Tree-LSTM | Topology — network structure matches parse tree | Recursive composition following tree edges |
| Graph NN | Connectivity — message passing follows graph edges | Adjacency matrix defines computation graph |
| HyperNetworks | Weights — parameters are generated per input | A generator network produces task-specific weights |
Dynamic Architecture is shape-shifting AI — models that physically reconfigure their computational structure to match the specific requirements of each input, moving beyond the rigid uniformity of static networks toward efficient, adaptive, input-aware computation.