Cross-Attention Variants

Cross-Attention Variants are modifications and extensions of the standard cross-attention mechanism — where queries come from one sequence and keys/values from another, used for conditioning, fusion, and multimodal interaction.

Key Variants

• Standard Cross-Attention: Decoder queries attend to encoder keys/values (original Transformer).
• Perceiver Cross-Attention: A small latent array cross-attends to a large input (bottleneck).
• Gated Cross-Attention: Cross-attention output is gated before adding to the residual (Flamingo).
• Multi-Source: Queries attend to multiple sources (e.g., text + image) with separate attention heads.
• Prompt Cross-Attention: Attend to a set of learned prompt tokens (parameter-efficient tuning).

Why It Matters

• Multimodal: Cross-attention is the primary mechanism for fusing information across modalities (text-image, text-audio).
• Conditioning: Used in diffusion models (Stable Diffusion) for text-conditioned image generation.
• Efficiency: Perceiver-style cross-attention enables processing arbitrarily large inputs through a fixed-size bottleneck.

Cross-Attention Variants are the bridges between sequences — the mechanism family that enables transformers to fuse, condition, and combine information across modalities.