Rrelu

RReLU (Randomized Leaky ReLU) is a variant of Leaky ReLU where the negative slope is randomly sampled from a uniform distribution during training — and fixed to the mean of that distribution during inference, providing built-in regularization.

Properties of RReLU

• Training: $\text{RReLU}(x) = \begin{cases} x & x > 0 \ a \cdot x & x \leq 0 end{cases}$ where $a \sim U(\text{lower}, \text{upper})$ (typically $U(0.01, 0.33)$).
• Inference: $a = (\text{lower} + \text{upper}) / 2$ (deterministic).
• Regularization: The randomness during training acts as a stochastic regularizer (similar to dropout).
• Paper: Xu et al. (2015).

Why It Matters

• Built-In Regularization: The random slope provides implicit regularization without explicit dropout.
• Kaggle: Popular in competition settings where every bit of regularization helps.
• Simplicity: No learnable parameters (unlike PReLU), but with regularization benefits.

RReLU is the stochastic ReLU — introducing randomness in the negative slope for built-in regularization during training.