MLP-Mixer for Vision

MLP-Mixer is the canonical all-MLP vision architecture that alternates token-mixing and channel-mixing layers on patch embeddings - it demonstrates that global spatial interaction can be learned through transposed MLP operations without any attention mechanism.

What Is MLP-Mixer?

• Definition: A stack of residual blocks where one MLP mixes information across tokens and a second MLP mixes information across channels.
• Patch Backbone: Input image is patchified and linearly projected to a fixed embedding dimension.
• Two Mixer Axes: Token mixing handles spatial relationships, channel mixing handles semantic feature transformation.
• Classifier Head: Global average pooling plus linear layer predicts class probabilities.

Why MLP-Mixer Matters

• Conceptual Clarity: Separates spatial and feature computation into explicit stages.
• Strong Baseline: Competitive accuracy when trained with large data and modern augmentation.
• Efficient Kernels: Dominated by matrix multiplies that are easy to optimize.
• Research Utility: Provides a neutral comparison point versus ViT and ConvNet families.
• Transferability: Architecture extends to audio and multimodal tokens with minor adaptation.

Block Anatomy

Token-Mixing MLP:
- Operates on transposed tensor so each channel mixes across all tokens.
- Captures long range dependencies across the full image grid.

Channel-Mixing MLP:
- Operates per token across channel dimension.
- Expands and contracts features with nonlinearity.

Residual + Norm:
- Pre-norm residual design improves optimization in deeper variants.

How It Works

Step 1: Convert image to N patch tokens, each with C channels, then apply token-mixing MLP across N for each channel independently.

Step 2: Apply channel-mixing MLP across C for each token, stack many blocks, pool token outputs, and classify.

Tools & Platforms

• timm: Reference Mixer implementations and pretrained checkpoints.
• JAX and Flax: Common research stack for large Mixer pretraining.
• TensorRT: Accelerates inference for matmul heavy backbones.

MLP-Mixer is the foundational all-MLP design that proves spatial reasoning does not strictly require attention or convolution - its clean decomposition makes it one of the most instructive modern vision baselines.