Mish

Mish is a smooth, self-regularizing activation function defined as $f(x) = x \cdot \tanh(\text{softplus}(x))$ — combining the benefits of Swish-like self-gating with a bounded below property that provides implicit regularization.

Properties of Mish

• Formula: $\text{Mish}(x) = x \cdot \tanh(ln(1 + e^x))$
• Smooth: Infinitely differentiable everywhere.
• Non-Monotonic: Like Swish, has a slight negative region, allowing negative gradients.
• Self-Regularizing: The bounded-below property prevents activations from going too negative.
• Paper: Misra (2019).

Why It Matters

• YOLOv4: Default activation in YOLOv4 and YOLOv5, where it outperforms Swish and ReLU.
• Marginally Better: Often 0.1-0.3% better than Swish in practice, though results are architecture-dependent.
• Compute: Slightly more expensive than Swish due to the tanh(softplus()) composition.

Mish is the smooth, self-regulating activation — a carefully crafted nonlinearity that provides consistent marginal improvements in deep networks.