Queue-Based Contrastive Learning

Queue-Based Contrastive Learning is the MoCo-style approach where negative samples are maintained in a FIFO queue — new batch representations are enqueued while the oldest are dequeued, providing a large, consistent pool of negatives with controlled staleness.

How Does the Queue Work?

• Enqueue: After each forward pass, the current batch's key representations (from the momentum encoder) are added to the queue.
• Dequeue: The oldest entries are removed.
• Queue Size: Typically 4096-65536. Independent of batch size.
• Consistency: Momentum encoder (slowly updated EMA) ensures the queue entries are reasonably consistent.

Why It Matters

• Decoupling: Batch size can be small (256) while the effective number of negatives is large (65K).
• MoCo v1/v2: The queue is the key innovation of MoCo, enabling SOTA performance on standard GPUs.
• vs. SimCLR: SimCLR requires batch size 4096-8192 (needs many GPUs). MoCo achieves similar results with batch size 256 + queue.

Queue-Based Contrastive Learning is the conveyor belt of negatives — continuously refreshing a large pool of comparison samples for effective contrastive training on modest hardware.