Zero-Cost Proxies

Zero-Cost Proxies are metrics that estimate the performance of a neural architecture without any training — computed in a single forward/backward pass at initialization, enabling architecture ranking in seconds instead of hours.

What Are Zero-Cost Proxies?

• Examples:
• SynFlow: Sum of product of all parameters' absolute values (measures signal propagation).
• NASWOT: Log-determinant of the neural tangent kernel at initialization.
• GradNorm: Norm of gradients at initialization.
• Fisher: Fisher information of the network at initialization.
• Cost: One forward + one backward pass = seconds per architecture.

Why It Matters

• Speed: Evaluate 10,000 architectures in minutes (vs. days for one-shot, weeks for full training).
• Pre-Filtering: Use zero-cost proxies to prune the search space before expensive evaluation.
• Limitation: Correlation with trained accuracy is imperfect (0.5-0.8 Spearman rank), but improving.

Zero-Cost Proxies are instant architecture critics — predicting network performance at birth, before a single weight update.