Causal Mask

Causal masks prevent attention to future tokens in autoregressive transformer models, enabling left-to-right generation.

Purpose
- During training on sequences, ensure each position can only see previous positions.
- Prevents information leakage from future tokens.

Implementation
- Lower triangular matrix of 1s, upper triangle masked with large negative values.
- Position i can attend to positions 0 to i, not i+1 onwards.

Why autoregressive
- Language generation is sequential, each token depends only on previous tokens.
- Model must learn to predict without seeing answer.

Training Efficiency
- Train on full sequence in parallel (teacher forcing) while maintaining causal constraint through masking.

Inference
- Not strictly needed (only past tokens exist), but often kept for consistency.

Combined with Padding
- Combine causal mask with padding mask for batched training.

KV Cache
- At inference, causal property enables KV caching since past representations don't change.

Decoder-only Models
- GPT, LLaMA, and most LLMs use causal masking throughout.