Neural Architecture Search

Neural Architecture Search (NAS) — using algorithms to automatically discover optimal neural network architectures instead of relying on human design, a key branch of AutoML.

The Problem

• Architecture design is manual and requires expert intuition
• Huge design space: Number of layers, filter sizes, connections, attention heads, activation functions
• Humans can't explore all possibilities

Search Strategies

• Reinforcement Learning NAS: A controller network proposes architectures; reward = validation accuracy. Original method (Google, 2017). Cost: 800 GPU-days
• Evolutionary NAS: Mutate and evolve a population of architectures. Similar cost to RL approach
• Differentiable NAS (DARTS): Make architecture choices continuous and differentiable → use gradient descent to search. Cost: 1-4 GPU-days (1000x cheaper)
• One-Shot NAS: Train a single supernet containing all candidate architectures, then extract the best subnet

Notable Results

• NASNet: Found architectures better than human-designed ResNet
• EfficientNet: NAS-designed CNN that set ImageNet records
• MnasNet: NAS for mobile — Pareto-optimal speed vs accuracy

Limitations

• Search space must be carefully defined by humans
• Results often aren't dramatically better than well-designed manual architectures
• Reproducibility challenges

NAS demonstrated that machines can design neural networks — but the community has shifted toward scaling known architectures rather than searching for new ones.