Poolformer

PoolFormer is the MetaFormer style architecture that replaces attention with simple pooling as token mixer while retaining strong residual transformer like skeleton - it argues that the block framework can be more important than the specific mixing operator.

What Is PoolFormer?

• Definition: A backbone built from MetaFormer blocks where token mixing is done by local pooling rather than attention.
• MetaFormer Template: Norm, token mixer, residual, channel MLP, residual.
• Simple Mixer: Average pooling layer with small kernel handles spatial interaction.
• Goal: Validate whether expensive attention is necessary for strong performance.

Why PoolFormer Matters

• Architectural Insight: Demonstrates value of block organization and optimization recipe.
• Efficiency: Pooling is cheaper and easier to optimize than attention.
• Stable Training: Simple operators reduce numerical complexity and training instability.
• Deployment Ready: Pooling kernels are universally supported across accelerators.
• Research Baseline: Useful for testing new ideas without heavy attention overhead.

PoolFormer Block Design

Token Mixer:
- Local average pooling injects neighborhood information.
- No dynamic attention weights are computed.

Channel MLP:
- Expands and contracts feature channels for semantic transformation.
- Often uses GELU activation and dropout.

Residual Structure:
- Two residual paths preserve gradient flow and depth scalability.

How It Works

Step 1: Patch embedding creates token map, then pooling mixer applies local spatial aggregation inside MetaFormer block.

Step 2: Channel MLP refines features, repeated blocks build hierarchy, and final pooled representation is classified.

Tools & Platforms

• timm: Reference PoolFormer implementations.
• PyTorch mobile: Efficient inference due to common pooling and MLP ops.
• Benchmark suites: Good baseline for comparing custom token mixers.

PoolFormer is a minimal yet strong proof that a well designed block scaffold can unlock performance even with very simple mixing operations - it is a practical and insightful baseline for efficient vision research.