Convmixer

ConvMixer is the patch based convolutional architecture that keeps ViT style patch embedding but uses depthwise and pointwise convolutions for mixing - it demonstrates that much of the performance gain comes from patch tokenization and modern training recipes, not only from attention.

What Is ConvMixer?

• Definition: A model that starts with patch embedding convolution, then repeats depthwise convolution for spatial mixing and pointwise convolution for channel mixing.
• Patch First Design: Treats image as coarse tokens from the first layer, similar to ViT patchify stage.
• Convolutional Mixer: Uses separable convolutions instead of attention for token interaction.
• Residual Blocks: Includes skip connections and activation normalization for stable deep training.

Why ConvMixer Matters

• Fair Comparison: Shows how strong patchification plus recipe tuning can make simple conv models highly competitive.
• Hardware Practicality: Convolution kernels are mature and highly optimized on many platforms.
• Data Efficiency: Often trains well on moderate data compared with data hungry transformer baselines.
• Interpretability: Depthwise filters are easier to inspect than dense attention weights.
• Deployment Speed: Inference stacks for conv operators are widely available and optimized.

ConvMixer Building Blocks

Patch Embedding Layer:
- Large stride convolution converts raw pixels into patch tokens.
- Sets token granularity and compute budget.

Depthwise Spatial Mixing:
- Per-channel spatial convolution captures local structure.
- Repeated blocks expand receptive field with depth.

Pointwise Channel Mixing:
- One by one convolution fuses channel information.
- Acts similarly to channel MLP in Mixer models.

How It Works

Step 1: Apply patch embedding convolution to convert image into low resolution token feature map.

Step 2: Repeat depthwise plus pointwise conv blocks with residual paths, then global pool and classify.

Tools & Platforms

• timm: Ready to use ConvMixer models and checkpoints.
• TensorRT and OpenVINO: Excellent support for separable conv inference.
• PyTorch: Straightforward to tune patch size, depth, and width.

ConvMixer is a strong reminder that patch tokenization and training strategy can rival more complex attention models - it offers a practical high speed baseline with familiar convolution operators.