Synthesizer

Synthesizer is a transformer variant that generates attention weights without computing query-key dot products — "synthesizing" attention maps directly from the input or from learned parameters, questioning whether explicit pairwise comparisons are necessary.

How Does Synthesizer Work?

• Dense Synthesizer: $A = \text{softmax}(f(X))$ where $f$ is a feedforward network. Attention from content, no Q-K dot product.
• Random Synthesizer: $A = \text{softmax}(R)$ where $R$ is a learnable random matrix. No input dependence at all.
• Mixture: Combine dense, random, and standard dot-product attention.
• Paper: Tay et al. (2021).

Why It Matters

• Provocative: Random attention (no Q-K interaction!) performs surprisingly well on many benchmarks.
• Insight: Suggests that the specific pairwise token comparison in standard attention may not always be necessary.
• Efficiency: Dense/random synthesizers can be faster than full dot-product attention.

Synthesizer is the experiment that questioned attention — showing that attention weights can be generated without even comparing tokens to each other.