Contrastive Decoding

Contrastive decoding is the decoding approach that selects tokens by contrasting scores from a strong model and a weaker reference model to discourage generic or low-quality continuations - it aims to improve coherence and specificity in generation.

What Is Contrastive decoding?

• Definition: Token ranking method based on score differences between expert and reference model outputs.
• Core Principle: Prefer tokens where the stronger model is confident but weaker model is less supportive.
• Quality Effect: Tends to suppress bland high-frequency continuations.
• Computation Requirement: Needs two-model scoring or equivalent contrastive signals during decoding.

Why Contrastive decoding Matters

• Text Quality: Can improve informativeness and reduce generic repetitive phrasing.
• Fluency Preservation: Maintains strong-model guidance while filtering weak continuations.
• Hallucination Mitigation: Contrastive signals may discourage unstable low-confidence branches.
• Task Benefit: Useful for detailed explanations and structured long responses.
• Research Relevance: Provides alternative to pure likelihood-based ranking criteria.

How It Is Used in Practice

• Reference Model Choice: Select a smaller or weaker model with compatible tokenization and domain behavior.
• Weight Calibration: Tune contrastive strength to balance specificity and grammatical stability.
• Ablation Testing: Evaluate repetition, relevance, and factuality against baseline decoding.

Contrastive decoding is a quality-oriented alternative to standard likelihood decoding - contrastive scoring can produce more informative outputs when tuned for stability.