Gradient-Based Nas

Gradient-Based NAS is a family of NAS methods that reformulate the architecture search as a continuous optimization problem — making architecture parameters differentiable and optimizable via gradient descent, dramatically reducing search cost compared to RL or evolutionary approaches.

How Does Gradient-Based NAS Work?

• Continuous Relaxation: Replace discrete architecture choices with continuous weights (softmax over operations).
• Bilevel Optimization: Alternately optimize architecture weights $alpha$ and network weights $w$.
• Methods: DARTS, ProxylessNAS, FBNet, SNAS.
• Speed: 1-4 GPU-days vs. 1000+ for RL-based methods.

Why It Matters

• Efficiency: Orders of magnitude faster than RL or evolutionary NAS.
• Simplicity: Standard gradient descent — no specialized RL or EA machinery needed.
• Challenges: Architecture collapse, weight entanglement, and the gap between continuous relaxation and discrete final architecture.

Gradient-Based NAS is turning architecture search into gradient descent — the insight that made neural architecture search practical for everyday use.