Convmixer
ConvMixer is a minimalist vision architecture that uses only standard depthwise separable convolutions for both spatial mixing and channel mixing, demonstrating that the "patching" strategy (dividing images into non-overlapping patches) introduced by Vision Transformers—not the attention mechanism—is a key ingredient for strong performance. ConvMixer applies a large-kernel depthwise convolution for spatial mixing and a pointwise (1×1) convolution for channel mixing, achieving competitive accuracy with extreme architectural simplicity.
Why ConvMixer Matters in AI/ML:
ConvMixer demonstrated that patch embedding is the critical innovation from ViTs, not self-attention, and that even simple convolutional architectures can match ViT performance when they adopt the same patch-based input processing strategy.
• Patch embedding — Like ViT and MLP-Mixer, ConvMixer first divides the input image into non-overlapping patches using a large-stride convolution (kernel=patch_size, stride=patch_size); this aggressive downsampling is the shared innovation across modern architectures
• Depthwise convolution — Spatial mixing uses depthwise convolution with large kernels (7×7 to 9×9): each channel is convolved independently, providing local spatial interaction without mixing channel information; this replaces both attention and MLP-based token mixing
• Pointwise (1×1) convolution — Channel mixing uses standard 1×1 convolutions that mix information across channels independently per spatial location, equivalent to a per-patch linear layer; this is the simplest possible channel interaction
• Isotropic design — Like ViT and MLP-Mixer, ConvMixer uses a uniform resolution throughout the network (no downsampling pyramid), processing patch tokens at constant spatial resolution through all layers
• Simplicity as a feature — ConvMixer has only three hyperparameters beyond depth: patch size, hidden dimension, and kernel size; this extreme simplicity makes it an ideal baseline for understanding which architectural components truly matter
| Component | ConvMixer | ViT | MLP-Mixer | ResNet |
|---|---|---|---|---|
| Patch Embedding | Conv (large stride) | Linear projection | Linear projection | None (gradual) |
| Spatial Mixing | Depthwise conv | Self-attention | Cross-patch MLP | 3×3 conv |
| Channel Mixing | 1×1 conv | FFN | Per-patch MLP | 1×1 conv |
| Resolution | Isotropic | Isotropic | Isotropic | Pyramidal |
| Inductive Bias | Local (conv kernel) | Global (attention) | Global (dense MLP) | Local (conv) |
| ImageNet Top-1 | 80-81% | 79-81% | 76-78% | 79-80% |
ConvMixer is the minimalist proof that the patch embedding strategy—not attention—is the transformative innovation from Vision Transformers, demonstrating that simple depthwise convolutions with aggressive patch-based input processing achieve competitive image classification accuracy with extreme architectural simplicity.