Random Synthesizer

Random Synthesizer is a variant of the Synthesizer model where attention weights are drawn from a learned matrix that is entirely independent of the input — each position in the attention map is a fixed, learned parameter, not computed from query-key interactions.

How Does Random Synthesizer Work?

• Attention Matrix: $A = \text{softmax}(R)$ where $R \in \mathbb{R}^{N \times N}$ is a learnable parameter matrix.
• No Input Dependence: The attention pattern is the same regardless of the input sequence.
• Training: $R$ is optimized via backpropagation alongside all other parameters.
• Fixed at Inference: Once trained, the attention pattern is static.

Why It Matters

• Surprising Result: Achieves 90%+ of dot-product attention performance on many tasks despite being input-independent.
• Implications: Suggests that transformers may partially rely on learned positional routing rather than semantic matching.
• Research Insight: Challenges the assumption that dynamic, content-based attention is essential.

Random Synthesizer is the attention paradox — showing that even fixed, input-independent attention patterns can capture surprisingly useful information.