Attention Bias Addition

Attention bias addition is the injection of structured bias terms into attention logits to encode positional or task priors before softmax - it influences which token relationships are favored without changing core attention mechanics.

What Is Attention bias addition?

• Definition: Adding learned or fixed bias values to QK score matrices prior to normalization.
• Common Forms: Relative position bias, ALiBi slopes, segment bias, and task-specific masking bias.
• Placement: Applied after raw score computation and before softmax scaling or normalization.
• Kernel Concern: Efficient implementations fuse bias injection with score computation.

Why Attention bias addition Matters

• Model Expressiveness: Encodes inductive structure that helps learning sequence relationships.
• Long-Range Behavior: Relative biases improve extrapolation for longer contexts in many settings.
• Task Adaptation: Domain-specific bias terms can improve performance for structured inputs.
• Runtime Cost: Naive bias handling can create extra memory movement and kernel launches.
• Optimization Opportunity: In-kernel bias addition preserves speed while retaining modeling benefits.

How It Is Used in Practice

• Bias Strategy: Choose fixed versus learned bias based on architecture and generalization goals.
• Fused Execution: Integrate bias math into fused attention kernels to minimize overhead.
• Ablation Testing: Measure quality gain and latency impact across sequence lengths.

Attention bias addition is a powerful control point in attention design - when implemented efficiently, it adds structural priors with minimal performance penalty.