Electra Generator-Discriminator

ELECTRA is a pre-training method that uses a generator-discriminator setup (inspired by GANs) for more sample-efficient language model pre-training — instead of predicting masked tokens (like BERT), ELECTRA trains a discriminator to detect which tokens in a sequence have been replaced by a small generator model.

ELECTRA Architecture

• Generator: A small masked language model that replaces [MASK] tokens with plausible alternatives.
• Discriminator: The main model — a Transformer that predicts whether EACH token is original or replaced.
• Binary Classification: Every token position provides a training signal — "original" or "replaced."
• Efficiency: The discriminator is trained on ALL tokens (not just the 15% masked) — 100% of positions provide signal.

Why It Matters

• Sample Efficiency: ELECTRA learns from every token position — ~4× more compute-efficient than BERT for the same performance.
• Small Models: Especially beneficial for small models — ELECTRA-Small outperforms GPT, BERT-Small by large margins.
• Replaced Token Detection: The RTD objective is more informative than MLM — learning to distinguish subtle corruptions.

ELECTRA is spot the fake token — a sample-efficient pre-training method that trains on every token position using replaced token detection.