Groupnorm

GroupNorm is a normalization technique that divides channels into groups and normalizes within each group — independent of batch size, making it the preferred normalization for tasks with small batch sizes (detection, segmentation, video).

How Does GroupNorm Work?

• Groups: Divide $C$ channels into $G$ groups of $C/G$ channels each (typically $G = 32$).
• Normalize: Compute mean and variance within each group (across spatial + channels-in-group dimensions).
• Affine: Apply learnable scale and shift per channel.
• Paper: Wu & He (2018).

Why It Matters

• Batch-Independent: Unlike BatchNorm, GroupNorm's statistics don't depend on batch size. Works with batch size 1.
• Detection/Segmentation: Standard in Mask R-CNN, DETR, and other detection frameworks where batch sizes are tiny (1-4).
• Special Cases: GroupNorm with $G = C$ is InstanceNorm. GroupNorm with $G = 1$ is LayerNorm.

GroupNorm is normalization for small batches — computing statistics within channel groups instead of across the batch for batch-size-independent training.