Neural Architecture Search Nas

Neural Architecture Search (NAS) is the automated machine learning technique that algorithmically discovers optimal neural network architectures for a given task — replacing manual architecture design with systematic exploration of topology, layer types, connectivity patterns, and hyperparameters to find designs that outperform human-designed networks.

Search Space Design:
- Cell-Based Search: define a DAG cell structure with learnable operations on each edge — discovered cell is stacked/repeated to build full network; reduces search space from exponential (full network) to manageable (single cell with ~10 edges)
- Operation Candidates: each edge can be one of K operations — typical choices: 3×3 conv, 5×5 conv, dilated conv, depthwise separable conv, max pool, avg pool, skip connection, zero (no connection)
- Macro Search: directly search for full network topology including layer count, widths, and skip connections — larger search space but can discover fundamentally novel architectures
- Hierarchical Search: search at multiple granularities — inner cell structure, cell connectivity, and network-level design (number of cells, reduction placement) each searched at appropriate level

Search Strategies:
- Reinforcement Learning (NASNet): controller RNN generates architecture descriptions, trained with REINFORCE using validation accuracy as reward — found NASNet achieving state-of-the-art ImageNet accuracy but required 48,000 GPU-hours
- Evolutionary (AmoebaNet): maintain population of architectures, mutate best performers, evaluate offspring — tournament selection with aging removes stagnant individuals; comparable to RL-based search at similar compute cost
- Differentiable (DARTS): relax discrete architecture choices to continuous weights over all operations — optimize architecture parameters via gradient descent simultaneously with network weights; reduces search from thousands of GPU-hours to single GPU-day
- One-Shot/Supernet: train a single overparameterized network containing all candidate operations — individual architectures are sub-networks evaluated by inheriting weights from the supernet; enables evaluating thousands of architectures without training each from scratch

Efficiency Improvements:
- Weight Sharing: all architectures in the search space share weights from a common supernet — eliminates the need to train each candidate independently; reduces search cost by 1000×
- Predictor-Based: train a performance predictor (neural network or Gaussian process) on evaluated architectures — use predictor to score unseen architectures without expensive training; focuses evaluation on promising candidates
- Hardware-Aware NAS: include latency, FLOPs, or energy as objectives alongside accuracy — multi-objective optimization produces Pareto-optimal architectures balancing accuracy with deployment constraints
- Zero-Cost Proxies: estimate architecture quality at initialization (before training) using gradient statistics — enables evaluating millions of candidates in minutes; examples include synflow, NASWOT, and jacob_cov scores

Neural Architecture Search represents the automation of the last major manual component in deep learning pipelines — while early NAS methods required enormous compute budgets, modern efficient NAS techniques discover architectures in hours that match or exceed years of expert human design effort.