ALiBi (attention with Linear Biases)
ALiBi (Attention with Linear Biases) is a positional encoding method for Transformers that replaces learned or sinusoidal positional embeddings with a simple linear penalty added directly to attention scores, where the penalty is proportional to the distance between the query and key tokens. ALiBi adds a bias of -m·|i-j| to the attention logit between positions i and j, where m is a fixed, head-specific slope that varies geometrically across attention heads.
Why ALiBi Matters in AI/ML:
ALiBi enables superior length extrapolation compared to other positional encodings, allowing models trained on short sequences to generalize to much longer sequences at inference time with minimal performance degradation, addressing a critical limitation of standard positional encodings.
• Linear distance penalty — The attention score becomes softmax(q_i^T·k_j - m·|i-j|), where the linear bias penalizes attending to distant tokens; this implements a soft local attention window whose effective width varies across heads due to different slope values m
• Head-specific slopes — Slopes are set to geometric sequence m_h = 1/2^(h·8/H) for H heads (e.g., for 8 heads: 1/2, 1/4, 1/8, ..., 1/256); heads with large slopes focus on nearby tokens (local patterns), while heads with small slopes attend to distant tokens (global patterns)
• Zero additional parameters — ALiBi requires no learned parameters for position encoding: slopes are fixed constants, and no positional embeddings are added to input tokens; this simplifies the model and reduces memory usage
• Length extrapolation — Models trained with ALiBi on sequences of length L can effectively process sequences of 2-4× L at inference time with graceful degradation, because the linear bias provides a smooth inductive bias for unseen distances rather than undefined embeddings
• No position embeddings — Unlike sinusoidal, learned, or RoPE encodings that modify token representations, ALiBi operates entirely in the attention logit space; input tokens are position-agnostic, and all positional information is injected at the attention computation
| Property | ALiBi | RoPE | Sinusoidal | Learned |
|---|---|---|---|---|
| Parameters | 0 | 0 | 0 | pos × d |
| Where Applied | Attention logits | Q,K vectors | Input embeddings | Input embeddings |
| Extrapolation | Excellent (2-4× L) | Moderate | Poor | None |
| Local vs Global | Multi-scale (per head) | Frequency-based | Frequency-based | Learned |
| Implementation | Add bias matrix | Rotate Q,K | Add to embeddings | Lookup table |
| Adopted By | BLOOM, MPT, Falcon | LLaMA, Mistral, PaLM | Original Transformer | BERT, GPT-2 |
ALiBi is the simplest and most effective method for achieving length extrapolation in Transformers, replacing complex positional embeddings with a parameter-free linear attention bias that provides multi-scale distance awareness across heads and enables models to generalize to sequence lengths far beyond their training context.