Neural Architecture Search Nas
Neural Architecture Search (NAS) is the automated process of discovering optimal neural network architectures by searching over a defined space of possible layer types, connections, and hyperparameters — replacing manual architecture design with algorithmic optimization that has produced architectures matching or exceeding human-designed networks on image classification, detection, and language tasks.
Search Space Design:
- Cell-Based Search: search for optimal cell (small computational block) and stack cells into full architecture; normal cells preserve spatial dimensions, reduction cells downsample; dramatically reduces search space vs searching full architectures directly
- Operations: candidate operations within each cell edge: convolution (3×3, 5×5, depthwise separable), pooling (max, avg), skip connection, zero (no connection); each edge selects one operation from the candidate set
- Macro Architecture: number of cells, channel width schedule, and cell connectivity are either fixed (cell-based NAS) or searched (hierarchical NAS); macro search is more flexible but exponentially larger search space
- Hardware-Aware Search: search space constrained by target hardware (latency, memory, FLOPs); lookup tables mapping operations to measured latency on target device enable hardware-aware objective optimization
Search Strategies:
- Reinforcement Learning NAS: controller (RNN) generates architecture description as sequence of tokens; architecture is trained and evaluated; reward (validation accuracy) updates the controller via REINFORCE; Zoph & Le (2017) original approach — effective but requires thousands of GPU-hours
- DARTS (Differentiable NAS): relaxes discrete architecture choices to continuous weights using softmax over operations on each edge; jointly optimizes architecture weights (which operations to keep) and network weights (operation parameters) via gradient descent; 1-4 GPU-days vs thousands for RL-NAS
- One-Shot NAS (Supernet): train a single supernet containing all possible architectures; evaluate candidate architectures by inheriting supernet weights; search reduces to selecting paths through the pretrained supernet — decouples training from search, enabling millions of architecture evaluations
- Evolutionary NAS: population of architectures mutated (change operations, add/remove connections) and evaluated; tournament selection retains best performers; naturally parallelizable across many GPUs; AmoebaNet achieved SOTA on ImageNet
Efficiency Improvements:
- Weight Sharing: all architectures in the search space share weights; avoids training each candidate from scratch; supernet training cost equivalent to training one large network — 1000× cheaper than independent training
- Proxy Tasks: evaluate architectures on smaller datasets (CIFAR-10 instead of ImageNet), fewer epochs (50 instead of 300), or reduced channel widths; rankings transfer approximately across scales for relative architecture comparison
- Predictor-Based Search: train a neural predictor that estimates architecture accuracy from its encoding; enables rapid evaluation of millions of candidates without actual training; predictors trained on hundreds of fully-evaluated architectures
- Zero-Cost Proxies: score architectures at initialization (no training) using gradient signals, Jacobian statistics, or linear region counts; 10000× faster than training-based evaluation but less reliable for fine-grained architecture ranking
Notable Discoveries:
- EfficientNet: compound scaling of depth, width, and resolution discovered by NAS; EfficientNet-B0 to B7 family achieved SOTA ImageNet accuracy with significantly fewer parameters and FLOPs than prior architectures
- NASNet/AmoebaNet: among first NAS-discovered architectures competitive with human-designed networks; transferred from CIFAR-10 search to ImageNet by stacking discovered cells
- Once-for-All (OFA): single supernet supporting 10^19 subnets; extract specialized architectures for different hardware targets without retraining — deploy the same supernet to phone, tablet, and server
- Hardware-Optimal Architectures: NAS consistently discovers architectures that differ from human intuition — favoring asymmetric structures, unusual operation combinations, and hardware-specific optimizations invisible to manual design
Neural architecture search is the automation of the most creative aspect of deep learning engineering — systematically exploring architectural possibilities that human designers would never consider, producing hardware-efficient architectures that define the performance frontier for vision, language, and multimodal AI models.