Activation Function Zoo

Activation Function Zoo refers to the large and growing collection of activation functions available for neural networks — from the classic sigmoid and tanh to modern learnable variants like Swish, Mish, and GELU, each with different properties for gradient flow, performance, and computational cost.

The Major Families

• Classic: Sigmoid, Tanh — smooth but suffer from vanishing gradients.
• ReLU Family: ReLU, Leaky ReLU, PReLU, ELU, SELU — fast, sparse, but can die (zero gradients).
• Smooth Non-Saturating: Swish, Mish, GELU — smooth approximations to ReLU with better gradient properties.
• Learnable: PReLU, Maxout, PAU — parameters that adapt during training.
• Gated: GLU, SwiGLU, GeGLU — multiplicative gating for transformers.

Why It Matters

• Architecture-Dependent: The best activation varies by architecture (ReLU for CNNs, GELU for transformers, SwiGLU for LLMs).
• Subtle Impact: Activation choice affects convergence speed, final accuracy, and computational cost.
• No Universal Best: Despite decades of research, no single activation dominates all settings.

The Activation Zoo is the menagerie of nonlinearities — each species evolved for a different ecological niche in the deep learning ecosystem.