Parametric Activation Functions

Parametric Activation Functions are activation functions with learnable parameters that are optimized during training — allowing the network to discover the optimal nonlinearity for each layer, rather than relying on a fixed, hand-designed function.

Key Parametric Activations

• PReLU: Learnable negative slope $a$ in $max(x, ax)$.
• Maxout: Max of $k$ learnable linear functions.
• PAU (Padé Activation Unit): Learnable rational function $P(x)/Q(x)$ with polynomial numerator and denominator.
• Adaptive Piecewise Linear: Learnable breakpoints and slopes for piecewise linear functions.
• ACON: Learnable smooth approximation that interpolates between linear and ReLU.

Why It Matters

• Flexibility: Each layer can learn its own optimal nonlinearity, potentially outperforming any fixed activation.
• Overhead: Adds few extra parameters but can significantly impact performance.
• Research: Shows that the choice of activation function matters more than commonly assumed.

Parametric Activations are the adaptive nonlinearities — letting the network evolve its own activation functions during training.