Selu

SELU (Scaled Exponential Linear Unit) is a self-normalizing activation function that automatically maintains zero mean and unit variance of activations — with specific scale parameters ($lambda approx 1.0507$, $alpha approx 1.6733$) derived to create a fixed-point attractor for the activation statistics.

Properties of SELU

• Formula: $\text{SELU}(x) = \lambda \begin{cases} x & x > 0 \ \alpha(e^x - 1) & x \leq 0 end{cases}$
• Self-Normalizing: Activations converge to zero mean and unit variance, even without BatchNorm.
• Requires: Specific initialization (LeCun Normal) and standard feedforward architecture.
• Paper: Klambauer et al. (2017).

Why It Matters

• No BatchNorm Needed: Self-normalization eliminates the need for explicit normalization layers.
• Deep Feedforward: Enables training 100+ layer feedforward networks without BN.
• Limitation: Only works well with fully connected architectures and specific initialization.

SELU is the self-normalizing activation — a mathematically designed fixed point that keeps activations stable through arbitrarily deep networks.