Crossvit
CrossViT is the dual-branch transformer that processes fine- and coarse-grained patch streams simultaneously and lets them exchange context via cross-attention — one branch sees small patches for texture while the other sees larger patches for layout, and bi-directional attention ensures both scales collaborate before classification.
What Is CrossViT?
- Definition: A vision transformer architecture with two parallel encoders: a tiny-patch branch (e.g., 8×8) and a large-patch branch (e.g., 16×16), each with its own attention layers.
- Key Feature 1: Cross-attention modules allow the branches to query each other, blending high-resolution cues with low-resolution context.
- Key Feature 2: Branch outputs are merged through concatenation or addition before the classifier, preserving multi-scale richness.
- Key Feature 3: Each branch can have different depths and channel widths to maintain computational balance.
- Key Feature 4: Relative positional biases align tokens across scales.
Why CrossViT Matters
- Scale Robustness: Small patches catch fine texture while large patches capture object-level structure, helping classification and detection alike.
- Efficient Fusion: Rather than building a massive single branch, the model processes two smaller streams in parallel.
- Transfer Flexibility: Branch-specific heads allow fine-tuning one branch for a new task while keeping the other frozen.
- Interpretability: Attention maps reveal whether decisions rely on detail or layout, aiding visualization.
- Plugin Friendly: CrossViT modules can be inserted into existing ViT backbones to add multi-scale reasoning.
Branch Configurations
Balance Strategy:
- Keep total FLOPs constant by adjusting depth and width per branch.
- Assign more layers to the small-patch branch for detail representation.
Cross-Attn Frequency:
- Insert cross-attention every few layers to share information at key intervals.
- Skip early cross-attention to let each branch extract its own features first.
Hierarchical Merge:
- Combine branch tokens progressively before final classification to create a fused representation.
How It Works / Technical Details
Step 1: Each branch computes standard multi-head attention within its patch scale, producing encoded tokens of matching spatial sizes.
Step 2: Cross-attention modules treat one branch as queries and the other as keys/values and vice versa, enabling mutual conditioning. The fused tokens then proceed through feed-forward layers and eventual concatenation.
Comparison / Alternatives
| Aspect | CrossViT | Pyramid ViT | Single-scale ViT |
|---|---|---|---|
| Scales | Dual fixed | Multi-stage | Single |
| Fusion | Cross-attention | Concatenation/FPN | None |
| Parameter Count | Moderate | Higher | Lowest |
| Applications | Fine+coarse tasks | Detection, segmentation | Classification |
Tools & Platforms
- Hugging Face Transformers: Contains CrossVitModel and CrossVitForImageClassification.
- timm: Implements cross attention layers that can plug into standard ViTs.
- MMDetection: Allows CrossViT backbones for detection by exposing feature maps at both scales.
- Visualization suites: Tools like Captum reveal cross-attention weights between scales.
CrossViT is the elegant multi-resolution duet that lets detail and layout sing together without forcing a single branch to be both wide and deep — it mixes fine texture with anchoring context for resilient visual recognition.