Neural Module Composition

Neural Module Composition is the architectural paradigm where neural network layouts are dynamically assembled at inference time by selecting and connecting specialized computational modules based on the structure of the input query — enabling Visual Question Answering (VQA) systems to parse a natural language question into a symbolic program and then wire together the corresponding neural modules into a custom computation graph that executes against the visual input.

What Is Neural Module Composition?

• Definition: Neural Module Composition refers to Neural Module Networks (NMNs) and their descendants — models that maintain a library of specialized neural modules (e.g., "Locate," "Describe," "Count," "Compare") and compose them into question-specific computation graphs at inference time. Rather than processing all questions through a fixed architecture, each question generates a unique program that determines which modules execute in what order.
• Dynamic Assembly: A semantic parser analyzes the input question ("What color is the large sphere left of the cube?") and produces a symbolic program: `Describe(Color, Filter(Large, Relate(Left, Locate(Sphere), Locate(Cube))))`. The system retrieves the neural weights for each module and wires them into a custom feedforward network that processes the image.
• Module Library: Each module is a small neural network specialized for a specific visual reasoning operation — spatial filtering, attribute extraction, counting, comparison, or relationship detection. Modules are trained jointly across all questions, learning reusable visual primitives.

Why Neural Module Composition Matters

• Compositional Generalization: Fixed-architecture VQA models memorize question-answer patterns and fail on novel compositions. Module composition generalizes systematically — if "red" and "sphere" modules work individually, "red sphere" works automatically by composing them, even if that exact combination never appeared in training.
• Interpretability: The program trace provides a complete, human-readable explanation of the reasoning process. For "How many red objects are bigger than the blue cylinder?", the trace shows: Filter(red) → FilterBigger(Filter(blue) → Filter(cylinder)) → Count — each step is inspectable and verifiable.
• Data Efficiency: Because modules learn reusable primitives rather than holistic pattern matching, new concepts can be learned from fewer examples. A new color module can be trained on a handful of examples and immediately composed with all existing shape, size, and relation modules.
• Scalability: The number of answerable questions scales combinatorially with the module library size. Adding one new module (e.g., "Behind") immediately enables all compositions involving spatial behind-relations without retraining existing modules.

Key Architectures

Neural Module Composition is on-the-fly neural circuit compilation — building a custom computation graph for every input by assembling specialized modules into question-specific reasoning pipelines that generalize compositionally to novel combinations.