Barlow Twins

Barlow Twins is a self-supervised learning method that learns representations by enforcing the cross-correlation matrix of embeddings to approach the identity matrix — making the representation invariant to augmentations while avoiding redundancy between dimensions.

How Does Barlow Twins Work?

• Input: Two augmented views of each image, encoded into embeddings $Z_A$ and $Z_B$.
• Loss: Cross-correlation matrix $C_{ij} = \frac{\sum_b z_{b,i}^A z_{b,j}^B}{\sqrt{\sum_b (z_{b,i}^A)^2}\sqrt{\sum_b (z_{b,j}^B)^2}}$.
• Objective: Push diagonal elements toward 1 (invariance) and off-diagonal toward 0 (reduce redundancy).
• Inspiration: Neuroscientist Horace Barlow's redundancy-reduction hypothesis.

Why It Matters

• Simple: No momentum encoder, no memory bank, no asymmetric architectures.
• No Negatives: Like BYOL, avoids the need for explicit negative samples.
• Conceptual Elegance: Directly optimizes information-theoretic properties of the representation.

Barlow Twins is making features independent and informative — using a redundancy-reduction principle from neuroscience to learn powerful, non-degenerate representations.