Sandwich Transformer

Sandwich Transformer is a transformer variant that reorders self-attention and feedforward sublayers — placing attention sublayers in the middle of the network and feedforward sublayers at the top and bottom, creating a "sandwich" structure that improves perplexity.

How Does Sandwich Transformer Work?

• Standard Transformer: Alternating [Attention, FFN, Attention, FFN, ...].
• Sandwich: [FFN, FFN, ..., Attention, Attention, ..., FFN, FFN, ...].
• Reordering: Attention layers are concentrated in the middle, FFN layers at the boundaries.
• Paper: Press et al. (2020).

Why It Matters

• Free Improvement: Simply reordering sublayers (no new parameters) improves language modeling perplexity.
• Insight: Suggests that the standard alternating pattern may not be optimal.
• Architecture Search: Motivates searching over sublayer orderings, not just sublayer types.

Sandwich Transformer is transformer with rearranged layers — the surprising finding that putting attention in the middle and FFN at the edges improves performance for free.