Differentiable Architecture Search

DARTS (Differentiable Architecture Search) is a gradient-based NAS method that makes the architecture search differentiable — by relaxing the discrete architecture choice into a continuous optimization problem, enabling efficient search using standard gradient descent in orders of magnitude less time.

How Does DARTS Work?

• Mixed Operations: Each edge in the search graph has all possible operations running in parallel, weighted by architecture parameters $\alpha$.
• Softmax: $\bar{o}(x) = \sum_k \frac{\exp(\alpha_k)}{\sum_j \exp(\alpha_j)} \cdot o_k(x)$
• Bilevel Optimization: Alternate between optimizing architecture weights $\alpha$ and network weights $w$.
• Discretization: After search, select the operation with highest $\alpha$ on each edge.

Why It Matters

• Speed: 1-4 GPU-days vs. 1000+ GPU-days for RL-based NAS.
• Simplicity: Standard gradient descent — no RL controllers or evolutionary populations needed.
• Limitation: Prone to architecture collapse (all edges converge to skip connections or parameter-free ops).

DARTS is gradient descent for architecture design — searching the space of possible networks as smoothly as training the weights of a single network.