Pyramid Vision Transformer (PVT)
Pyramid Vision Transformer (PVT) is a hierarchical vision Transformer that introduces progressive spatial reduction across four stages, generating multi-scale feature maps similar to CNN feature pyramids while using self-attention as the core computation. PVT addresses ViT's two key limitations for dense prediction tasks: the lack of multi-scale features and the quadratic complexity of global attention on high-resolution feature maps.
Why PVT Matters in AI/ML:
PVT was one of the first pure Transformer backbones for dense prediction (detection, segmentation), demonstrating that Transformers can replace CNNs as general-purpose visual feature extractors when designed with multi-scale output and efficient attention.
• Progressive spatial reduction — PVT processes features through four stages with spatial dimensions [H/4, H/8, H/16, H/32] and increasing channel dimensions [64, 128, 320, 512], producing a feature pyramid identical in structure to ResNet's C2-C5 stages
• Spatial Reduction Attention (SRA) — To handle the large number of tokens at early stages (high resolution), PVT reduces the spatial dimension of keys and values by a factor R before computing attention: K̃ = Reshape(K, R)·W_s, reducing complexity from O(N²) to O(N²/R²)
• Patch embedding between stages — Overlapping patch embedding layers (strided convolutions) between stages reduce spatial resolution by 2× while increasing channel dimension, serving the same role as pooling/striding in CNNs
• Dense prediction compatibility — PVT's multi-scale outputs plug directly into existing detection heads (Feature Pyramid Network, RetinaNet) and segmentation heads (Semantic FPN, UPerNet) designed for CNN feature pyramids
• PVTv2 improvements — PVT v2 replaced position embeddings with convolutional position encoding (zero-padding convolution), added overlapping patch embedding, and improved SRA with linear complexity attention, achieving better performance and flexibility
| Stage | Resolution | Channels | Tokens | SRA Reduction |
|---|---|---|---|---|
| Stage 1 | H/4 × W/4 | 64 | N/16 | R=8 |
| Stage 2 | H/8 × W/8 | 128 | N/64 | R=4 |
| Stage 3 | H/16 × W/16 | 320 | N/256 | R=2 |
| Stage 4 | H/32 × W/32 | 512 | N/1024 | R=1 |
| Output | Multi-scale pyramid | 64-512 | Multi-resolution | Scales with stage |
Pyramid Vision Transformer pioneered the hierarchical Transformer backbone for computer vision, demonstrating that multi-scale feature pyramids with spatially reduced attention enable pure Transformer architectures to serve as drop-in replacements for CNN backbones in detection, segmentation, and all dense prediction tasks.