Patch Merging

Patch Merging is a downsampling operation in Vision Transformers that reduces the number of tokens by merging adjacent patches — similar to strided convolution in CNNs, creating a hierarchical representation with progressively fewer, richer tokens.

How Does Patch Merging Work?

• Group: Take 2×2 groups of adjacent tokens (4 tokens per group).
• Concatenate: Concatenate their features along the channel dimension ($C → 4C$).
• Project: Linear projection to reduce channels ($4C → 2C$).
• Result: Spatial resolution halved (H/2 × W/2), channels doubled ($2C$).
• Used In: Swin Transformer, Twins, PVT.

Why It Matters

• Hierarchical ViT: Enables ViTs to have a multi-scale, pyramid-like structure similar to CNNs.
• Dense Prediction: The multi-scale feature maps are essential for detection and segmentation.
• Efficiency: Fewer tokens at later stages -> reduced attention computation.

Patch Merging is pooling for Vision Transformers — creating a multi-resolution feature hierarchy by progressively combining adjacent tokens.