Unidirectional Attention

Unidirectional attention restricts each token to attending only to previous tokens, enabling autoregressive generation. Mechanism: Causal mask applied, position i attends to positions 0 through i only. Same as GPT-style left-to-right attention. Why unidirectional: Generation is sequential, predict next token based only on past. Matches natural language production. Training: Can still train on full sequences using teacher forcing, but mask ensures each prediction uses only prior context. Advantages: Enables generation, simple and efficient, scales well. Disadvantages: Cannot use future context for understanding. Less effective for tasks requiring full sentence understanding. KV cache benefit: Since attention is only to past, key-value pairs can be cached and reused during generation. Huge speedup. Used by: GPT series, LLaMA, Claude, most production LLMs. Comparison: Bidirectional better for embeddings/understanding, unidirectional required for generation. Many approaches try to get benefits of both.