Celu

CELU (Continuously Differentiable Exponential Linear Unit) is a modification of ELU that ensures continuous first derivatives — addressing the non-differentiability of ELU at $x = 0$ when $\alpha
eq 1$ by using a scaled exponential formulation.

Properties of CELU

• Formula: $\text{CELU}(x) = \begin{cases} x & x > 0 \ \alpha(\exp(x/\alpha) - 1) & x \leq 0 end{cases}$
• $C^1$ Smoothness: Continuously differentiable everywhere, including at $x = 0$, for any $\alpha > 0$.
• Parameterized: $\alpha$ controls the saturation value and the smoothness for negative inputs.
• Paper: Barron (2017).

Why It Matters

- Mathematical Correctness: Fixes the differentiability issue of ELU when $\alpha
eq 1$.
- Optimization: Smooth activations generally lead to smoother loss landscapes and easier optimization.
- Niche: Less widely adopted than GELU/Swish but theoretically well-motivated.

CELU is the mathematically correct ELU — ensuring smooth differentiability for any choice of the saturation parameter.