Resmlp
ResMLP is a simplified all-MLP architecture for image classification that applies residual connections to a pure MLP design, using cross-patch linear layers for spatial interaction and per-patch MLPs for channel interaction, with Affine transformations replacing LayerNorm for normalization. ResMLP demonstrates that even simpler MLP architectures than MLP-Mixer can achieve competitive vision performance with appropriate training recipes.
Why ResMLP Matters in AI/ML:
ResMLP showed that extreme architectural simplicity—linear cross-patch layers without non-linearities for spatial mixing—can achieve strong image classification accuracy, further narrowing the gap between simple MLPs and sophisticated attention-based architectures.
• Cross-patch linear layer — Instead of MLP-Mixer's nonlinear token-mixing MLP, ResMLP uses a single linear layer (matrix multiplication) for spatial interaction: Y = X + A·X^T (transposed to mix across patches), eliminating nonlinearities from the spatial mixing step entirely
• Affine normalization — ResMLP replaces LayerNorm with a simpler Affine transformation: Aff(x) = α ⊙ x + β, where α and β are learnable per-channel parameters; this is computationally cheaper and avoids the batch/instance statistics of normalization
• Per-patch channel MLP — The channel-mixing component remains a standard two-layer MLP with GELU activation: FFN(x) = W₂·GELU(W₁·x + b₁) + b₂, applied independently to each patch, identical to the feed-forward layers in standard Transformers
• Residual connections — Skip connections around both the cross-patch layer and the channel MLP stabilize training and enable deeper networks; pre-normalization (Affine before each sublayer) follows the standard Transformer convention
• Training recipe importance — ResMLP's competitive performance depends heavily on modern training techniques: data augmentation (RandAugment, Mixup, CutMix), regularization (stochastic depth, dropout), and long training schedules (400+ epochs)
| Component | ResMLP | MLP-Mixer | ViT |
|---|---|---|---|
| Spatial Mixing | Linear (no nonlinearity) | MLP (with GELU) | Self-attention |
| Channel Mixing | MLP (GELU) | MLP (GELU) | MLP (GELU) |
| Normalization | Affine | LayerNorm | LayerNorm |
| Residual Connections | Yes | Yes | Yes |
| Parameters (Base) | ~100M | ~60M | ~86M |
| ImageNet Top-1 | 79.4% (ResMLP-B24) | 76.4% (Mixer-B/16) | 79.9% (ViT-B/16) |
| Training Recipes | Critical | Important | Important |
ResMLP pushes the simplicity frontier of vision architectures even further than MLP-Mixer, demonstrating that linear spatial mixing without any nonlinearity combined with per-patch feed-forward networks and modern training techniques can approach the accuracy of attention-based models, reinforcing the finding that training recipes and scale matter more than architectural sophistication.