Compound Scaling

Compound Scaling is the principled method of jointly scaling a neural network's depth, width, and resolution using a fixed ratio — introduced in EfficientNet, showing that balanced scaling outperforms scaling any single dimension.

How Does Compound Scaling Work?

• Three Dimensions: Depth ($d$), Width ($w$), Resolution ($r$).
• Constraint: $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$ (doubles FLOPs per unit increase in $\phi$).
• Grid Search: Find optimal $\alpha, \beta, \gamma$ on a small model (B0). Then scale with $\phi$.
• Result: $d = \alpha^\phi, w = \beta^\phi, r = \gamma^\phi$.

Why It Matters

• Balanced Growth: Networks that only grow deeper (ResNet-1000) or only wider (Wide-ResNet) hit diminishing returns. Compound scaling avoids this.
• Universal: The principle applies beyond EfficientNet — any architecture benefits from balanced scaling.
• Design Rule: Provides a concrete recipe for scaling any base architecture.

Compound Scaling is the growth formula for neural networks — a mathematical recipe ensuring balanced development across all dimensions.