Attention Mask

Attention masks indicate which tokens the model should attend to versus ignore during self-attention computation. Purpose: Prevent attention to padding tokens, mask future tokens in causal models, handle variable-length sequences in batches. Format: Binary tensor same shape as input, 1 = attend, 0 = ignore. Applied as additive mask (large negative value) to attention scores before softmax. Padding mask: Mask out PAD tokens so they dont influence representations. Essential for batched inference with different sequence lengths. For training: Prevents padding from affecting gradients, ensures loss computed only on real tokens. Creation: Usually automatic from tokenizer when padding. Can be manually constructed for custom masking. Multi-head attention: Same mask typically applied across all attention heads. Cross-attention: May have different masks for encoder and decoder sequences. Debugging: Incorrect attention masks cause subtle bugs, degraded performance, or training instability. Always verify mask shapes and values.