Hourglass Transformer

Hourglass Transformer is an efficient transformer that uses a U-Net-like architecture — first downsampling the sequence (reducing token count), processing at reduced resolution, then upsampling back, with skip connections preserving fine-grained information.

How Does Hourglass Transformer Work?

• Downsample: Reduce sequence length via pooling or strided operations.
• Process: Apply transformer blocks at the reduced resolution (cheaper attention).
• Upsample: Restore original sequence length via interpolation or transposed operations.
• Skip Connections: Concatenate or add features from the downsampling path to the upsampling path.
• Paper: Nawrot et al. (2022).

Why It Matters

• U-Net Success: Brings the highly successful U-Net architecture pattern from vision to sequence modeling.
• Efficiency: Most computation happens at reduced resolution -> significant speedup for long sequences.
• Quality: Skip connections preserve fine-grained token-level information despite the compression.

Hourglass Transformer is U-Net meets transformers — compressing, processing, and expanding sequences with skip connections for efficient long-range modeling.