Gmlp for Vision

gMLP is the gated MLP architecture that injects spatial interaction through a Spatial Gating Unit while keeping the model attention free - it multiplies one feature branch by a learned spatial projection of another branch, creating content-aware modulation without softmax attention.

What Is gMLP?

• Definition: An MLP based block that splits channels, processes one half through a spatial projection, and gates the other half.
• Spatial Gating Unit: Central mechanism that enables token level interaction across sequence positions.
• Residual Design: Standard residual wrappers keep training stable in deeper stacks.
• Flexibility: Can be used in pure all-MLP backbones or hybridized with convolution and attention blocks.

Why gMLP Matters

• Content Modulation: Gating introduces adaptive behavior beyond plain linear token mixing.
• Lower Overhead: Avoids quadratic attention maps and reduces memory pressure.
• Strong Baseline: Competitive performance in classification with tuned recipes.
• Hybrid Utility: Useful as a drop-in block for efficient backbones.
• Research Value: Helps isolate the benefit of gating versus explicit attention.

gMLP Block Structure

Channel Split:
- Input channels are divided into gating branch and value branch.
- Each branch receives separate linear transforms.

Spatial Projection:
- Gating branch is projected along token dimension to encode global context.
- Projection weights are learned end to end.

Elementwise Gate:
- Value branch is multiplied by projected gate signal.
- Output then passes through residual and normalization.

How It Works

Step 1: Patch embeddings enter gMLP block, channel split is performed, and gate branch is transformed across tokens.

Step 2: Gate output modulates value branch by elementwise multiplication, then residual addition and feedforward layers continue.

Tools & Platforms

• timm: gMLP variants for rapid benchmarking.
• PyTorch Lightning: Good for ablation on gate width and depth.
• Inference SDKs: Gate operations map well to standard tensor kernels.

gMLP is an efficient middle ground between plain MLP mixing and full attention complexity - its spatial gating unit delivers adaptive context flow with a compact compute profile.