Softplus

Softplus is a smooth approximation to ReLU defined as $f(x) = ln(1 + e^x)$ — providing a continuously differentiable alternative that never outputs exactly zero, making it useful in contexts where strict positivity is required.

Properties of Softplus

• Formula: $\text{Softplus}(x) = ln(1 + e^x)$
• Derivative: $\text{Softplus}'(x) = \sigma(x)$ (the sigmoid function).
• Approximation: Closely approximates ReLU for large $|x|$. Smoother near zero.
• Strictly Positive: $\text{Softplus}(x) > 0$ for all $x$ (unlike ReLU which outputs 0 for $x \leq 0$).

Why It Matters

• Variance Modeling: Used as the output activation for predicting variance/scale parameters (must be positive).
• Theoretical: Connects ReLU to sigmoid through differentiation (Softplus → sigmoid → logistic).
• Building Block: Used inside other activations like Mish: $\text{Mish}(x) = x \cdot \tanh(\text{Softplus}(x))$.

Softplus is the smooth version of ReLU — a continuously differentiable, strictly positive function used where smoothness and positivity are essential.