Replaced Token Detection

RTD (Replaced Token Detection) is the pre-training objective used by ELECTRA — the model is trained to predict, for each token in a sequence, whether it is the original token or a replacement inserted by a generator model, providing a binary classification signal at every token position.

RTD Details

• Generator: A small masked LM replaces ~15% of tokens with plausible alternatives — "the" might be replaced with "a."
• Discriminator: Predicts $p(\text{original} | x_i, \text{context})$ for EVERY position $i$ — binary classification.
• All Positions: Training signal from 100% of positions (vs. 15% for MLM) — much more efficient.
• Subtle Corruptions: The generator produces plausible replacements — the discriminator must learn fine-grained language understanding.

Why It Matters

• Efficiency: 4× more sample-efficient than Masked Language Modeling — less data and compute for the same performance.
• Signal Density: Every token provides a training signal — no wasted computation on non-masked positions.
• Transfer: ELECTRA's discriminator transfers well to downstream tasks — competitive with or better than BERT.

RTD is real or fake at every position — a dense pre-training signal that makes language model training dramatically more sample-efficient.