Evonorm

EvoNorm is a family of normalization-activation layers discovered by automated search — using evolutionary algorithms to find novel combinations of normalization and activation operations that outperform hand-designed ones like BN-ReLU or GN-ReLU.

How Was EvoNorm Discovered?

• Search Space: Primitive operations (mean, variance, sigmoid, multiplication, max, etc.) combined in computation graphs.
• Objective: Maximize validation accuracy on ImageNet with various architectures.
• Results: EvoNorm-B0 (batch-dependent, replaces BN-ReLU), EvoNorm-S0 (batch-independent, replaces GN-ReLU).
• Paper: Liu et al. (2020).

Why It Matters

• Beyond Hand-Design: Demonstrates that automated search can discover normalization layers humans haven't considered.
• Performance: EvoNorm-S0 matches BatchNorm+ReLU accuracy while being batch-independent.
• Joint Design: Searches normalization and activation together, finding synergies that separate design misses.

EvoNorm is evolved normalization — normalization-activation layers discovered by evolution rather than human intuition.