Memory Bank

Memory Bank is a data structure used in contrastive self-supervised learning to store a large collection of negative sample representations — enabling effective contrastive learning with small batch sizes by decoupling the number of negatives from the batch size.

What Is a Memory Bank?

• Structure: A dictionary/queue storing feature vectors from previous forward passes.
• Size: Typically 4K-65K entries (much larger than a single batch).
• Update: Features are computed with the current encoder and stored. Older entries are replaced (FIFO or random).
• Used By: MoCo (momentum-updated queue), InstDisc (full memory bank).

Why It Matters

• GPU Efficiency: Small batches fit on any GPU, but the memory bank provides thousands of negatives for the contrastive loss.
• Staleness Trade-off: Stored features were computed by an older version of the encoder -> stale representations.
• MoCo Solution: Uses a slowly-updated momentum encoder to reduce staleness.

Memory Bank is the archive of past representations — a clever trick that provides a large, diverse pool of negatives without requiring massive batch sizes.