Infonce Loss

InfoNCE Loss is a contrastive learning objective that estimates mutual information between representations — by training a model to identify the correct "positive" sample from a set of "negative" distractors, forming the core loss function behind CPC, MoCo, and SimCLR.

What Is InfoNCE?

• Formula: $\mathcal{L} = -\log \frac{\exp(\sim(z_i, z_j^+)/\tau)}{\sum_{k=0}^{K} \exp(\sim(z_i, z_k)/\tau)}$
• Positive Pair ($z_i, z_j^+$): Two augmented views of the same sample.
• Negatives ($z_k$): All other samples in the batch (or memory bank).
• Temperature ($\tau$): Controls the sharpness of the distribution.

Why It Matters

• Foundation: The mathematical engine behind modern contrastive self-supervised learning.
• Mutual Information: Lower bound on the mutual information $I(X; Z)$ between input and representation.
• Scalability: Performance improves with more negatives (larger batch size or memory bank).

InfoNCE is the core loss function of contrastive learning — teaching representations by distinguishing the real match from thousands of imposters.