Neural Architecture Search (Nas)

Neural Architecture Search (NAS) automatically discovers optimal model architectures instead of manual design. Motivation: Architecture design requires expertise and intuition. Automate to find better architectures efficiently. Search space: Define possible operations (conv sizes, attention types), connectivity patterns, depth/width ranges. Search methods: Reinforcement learning: Controller network proposes architectures, trained on validation performance. Evolutionary: Population of architectures, mutate and select best. Gradient-based: Differentiable architecture, learn architecture parameters (DARTS). Weight sharing: Train supernet containing all possible architectures, evaluate subnets. Compute cost: Early NAS required thousands of GPU-days. Modern methods reduce to GPU-hours through weight sharing. Notable success: EfficientNet family found by NAS, outperformed manual designs. AmoebaNet, NASNet. For transformers: AutoML searches over attention patterns, FFN sizes, layer configurations. Search vs transfer: Once good architecture found, transfer to new tasks. NAS is research tool. Current status: Influential for initial architecture discovery, but recent trend toward scaling simple architectures (plain transformers) rather than complex search.