Token Labeling

Token Labeling is a training strategy for Vision Transformers that assigns individual labels to each patch token — rather than only supervising the CLS token, providing dense supervision that encourages every token to learn meaningful representations.

How Does Token Labeling Work?

• Teacher Model: A pre-trained CNN or ViT generates soft predictions for each spatial region.
• Token Labels: Each patch token receives a soft label from the teacher's corresponding spatial prediction.
• Dual Loss: $\mathcal{L} = \mathcal{L}_{CLS} + \alpha \cdot \mathcal{L}_{tokens}$ (supervise both CLS and individual tokens).
• Paper: Jiang et al. (2021, "All Tokens Matter").

Why It Matters

• Dense Supervision: Every token learns to be discriminative, not just the CLS token.
• Better Features: Token-level supervision produces better intermediate features for downstream tasks.
• Free Improvement: ~1% accuracy gain on ImageNet with no architectural changes.

Token Labeling is teaching every patch to recognize — providing individual supervision to each token for richer, more discriminative ViT features.