Decoder Only

Decoder-Only Transformer is the dominant architecture for large language models that processes input sequences left-to-right using causal (autoregressive) masking — generating tokens one at a time where each token can only attend to previous tokens in the sequence, unifying both "understanding" (processing the input prefix) and "generation" (producing new tokens) in a single model stack, as used by GPT-4, Claude, LLaMA, Gemini, and virtually all modern LLMs.

What Is a Decoder-Only Transformer?

• Definition: A transformer architecture consisting only of decoder blocks with causal self-attention — each position can attend to itself and all previous positions but not future positions, enforced by masking the upper triangle of the attention matrix with negative infinity before softmax.
• Autoregressive Generation: Tokens are generated one at a time, left to right — at each step, the model predicts the probability distribution over the vocabulary for the next token, samples or selects a token, appends it to the sequence, and repeats.
• Causal Masking: The attention mask ensures position i can only attend to positions 0 through i — this prevents "cheating" during training (the model can't look at future tokens it's supposed to predict) and enables efficient autoregressive generation at inference time.
• Unified Architecture: Unlike encoder-decoder models that separate understanding and generation, decoder-only models handle both in one stack — the input prompt is processed as a prefix (like an encoder), and generation continues from where the prefix ends.

Why Decoder-Only Dominates

• Scaling Laws: Decoder-only architectures have demonstrated the most predictable scaling behavior — performance improves smoothly and predictably with more parameters, data, and compute, as shown by the Chinchilla scaling laws.
• Simplicity: One model architecture, one training objective (next-token prediction), one inference procedure — simpler than encoder-decoder which requires separate encoder and decoder passes.
• KV Caching: During generation, the Key and Value matrices for all previous tokens can be cached and reused — only the new token's Q, K, V need to be computed at each step, making generation efficient.
• Prompting Flexibility: Input and output are just one continuous sequence — "understanding" tasks (classification, extraction) are handled by prompting, and "generation" tasks (writing, coding) are handled naturally.

Decoder-Only vs. Encoder-Decoder

Decoder-only transformers are the architecture powering the current generation of large language models — using causal masking and autoregressive generation to unify language understanding and generation in a single model that scales predictably to hundreds of billions of parameters, establishing the dominant paradigm for AI systems from chatbots to code generation to reasoning.