Causal Language Modeling

Causal Language Modeling (CLM), or autoregressive language modeling, is the pre-training objective where the model predicts the next token in a sequence conditioned ONLY on the previous tokens — used by the GPT family (GPT-2, GPT-3, GPT-4), it learns the joint probability $P(x) = \prod P(x_i | x_{<i})$.

CLM Characteristics

• Unidirectional: Attention is masked so position $i$ can only attend to $0 dots i-1$. No looking ahead.
• Generation: Naturally suited for text generation — predicts one token at a time.
• Efficiency: Very scalable pre-training (predict next word is a hard, dense signal).
• Limitation: Cannot use bidirectional context for understanding (finding the subject of a sentence that appears later).

Why It Matters

• Generative AI: The foundation of all modern generative LLMs (ChatGPT, Claude, LLaMA).
• Scale: Has been shown to scale incredibly well (Scaling Laws) — more data + compute = better reasoning.
• Zero-Shot: CLM models exhibit strong zero-shot and few-shot learning capabilities (in-context learning).

Causal Language Modeling is predicting the future — learning to guess the next word based on history, the engine of modern generative AI.