Gradient Centralization

Gradient Centralization (GC) is a simple optimization technique that centralizes (zero-means) gradients before each update — subtracting the mean of the gradient vector from each element, which acts as a regularizer and improves training stability and generalization.

How Does Gradient Centralization Work?

• Operation: For each weight tensor, compute $\hat{g} = g - \bar{g}$ where $\bar{g}$ is the column-wise mean.
• Constraint: The resulting update has zero mean -> constrains the weight space.
• Integration: Applied as a single line of code before the optimizer update step.
• Paper: Yong et al. (2020).

Why It Matters

• Simplicity: One line of code, no additional hyperparameters, works with any optimizer.
• Regularization: Acts as implicit regularization by constraining the update direction.
• Performance: Consistently improves both convergence speed and final accuracy by 0.1-0.5%.

Gradient Centralization is the zero-mean trick for gradients — a remarkably simple technique that improves training for free.