Contrastive Predictive Coding
Contrastive Predictive Coding (CPC) is a self-supervised representation learning method that trains neural encoders by predicting future observations in latent space using contrastive objectives — maximizing mutual information between a compact context representation and future encoded observations while distinguishing true futures from random negative samples — introduced by van den Oord et al. (DeepMind, 2018) as a unifying framework that simultaneously achieved state-of-the-art self-supervised representations for speech, images, text, and reinforcement learning, directly inspiring wav2vec, SimCLR, and the broader contrastive learning revolution.
What Is CPC?
- Core Idea: Learn representations that are maximally informative about the future by training a model to predict future latent codes from a context — without ever predicting raw pixels or audio waveforms.
- Encoder: Maps raw observations (audio frames, image patches, words) to latent representations z_t.
- Autoregressive Context Model: A recurrent network aggregates past representations into a context vector c_t, which summarizes the history up to time t.
- Prediction: Linear predictors W_k map context c_t to predicted future representations for k steps ahead: z_hat_{t+k} = W_k c_t.
- InfoNCE Loss: The model is trained to identify the true future z_{t+k} among N-1 randomly sampled "negative" representations from the same batch — a contrastive multi-class classification problem.
Why Predict in Latent Space?
- Avoids Modeling Irrelevant Details: Predicting raw waveforms or pixels is dominated by low-level statistics. Predicting latent codes focuses the model on semantically informative structure.
- Slow Features: Meaningful semantic content (speaker identity, object category, sentence meaning) changes more slowly than raw signal variations — latent prediction captures these slow features.
- Mutual Information Bound: The InfoNCE loss is a lower bound on I(z_{t+k}; c_t) — the mutual information between the context and the future. Maximizing InfoNCE maximizes predictive mutual information.
Influence on Self-Supervised Learning
| Method | How It Extends CPC |
|---|---|
| wav2vec 2.0 | CPC applied to quantized speech codes — foundation of modern ASR |
| SimCLR | Drops temporal structure; applies contrastive prediction to augmented image pairs |
| MoCo | Momentum encoder + memory bank for large negative sets — CPC scaled for vision |
| Data2Vec | Generalizes CPC's predictive coding idea across speech, vision, and language |
| CPC for RL (CURL, ATC) | Applies contrastive coding to RL state representations |
Applications
- Speech: CPC representations transfer to phoneme detection, speaker verification, and ASR without any labeled data — demonstrating that temporal predictability captures phonetic structure.
- Computer Vision: Predicting spatial patches from context in images learns features competitive with fully supervised models.
- Natural Language Processing: Temporal CPC on sentences learns bidirectional contextual representations.
- Reinforcement Learning: CPC state encoders improve sample efficiency dramatically in pixel-observation RL tasks.
Contrastive Predictive Coding is the self-supervised principle that the best representations are those that predict the future — the insight that learning to forecast in latent space extracts the structural regularities of the world, producing representations that transfer broadly across downstream tasks without a single manual label.