Filter Response Normalization

FRN (Filter Response Normalization) is a normalization technique designed to work without batch or group dependencies — normalizing each filter response individually and using a learnable thresholded linear unit (TLU) as the activation function.

How Does FRN Work?

• Normalize: $\hat{x}_c = x_c / \sqrt{\frac{1}{HW}\sum_{h,w} x_{c,h,w}^2 + \epsilon}$ (divide by RMS of spatial dimensions for each channel).
• TLU Activation: $y = \max(x, \tau)$ where $\tau$ is a learnable threshold (replaces ReLU).
• No Mean Subtraction: Like RMSNorm, FRN skips mean centering.
• Paper: Singh & Krishnan (2020).

Why It Matters

• Batch-Free: Works with batch size 1, unlike BatchNorm.
• SOTA: Achieved competitive results with BatchNorm across various CNN architectures.
• TLU: The learnable threshold activation is key — standard ReLU doesn't work well with FRN.

FRN is self-sufficient normalization — each filter channel normalizes itself independently, with a learnable activation threshold for optimal performance.