Class Token
Class token (CLS) is a special learnable embedding vector prepended to the sequence of patch tokens in a Vision Transformer that aggregates global image information through self-attention — serving as the summary representation of the entire image that is ultimately fed into the classification head to produce the final prediction.
What Is the Class Token?
- Definition: A trainable parameter vector of the same dimension as patch embeddings (e.g., 768-D for ViT-Base) that is concatenated to the beginning of the patch token sequence before being processed by the transformer encoder layers.
- Origin: Borrowed directly from BERT (Bidirectional Encoder Representations from Transformers), where the [CLS] token similarly aggregates sequence-level information for classification tasks.
- Sequence Position: Added as position 0, making the full input sequence [CLS, patch_1, patch_2, ..., patch_N] with length N+1 (e.g., 197 tokens for 196 patches + 1 CLS).
- Output Usage: After passing through all transformer layers, only the CLS token's final hidden state is used for classification — it is fed into an MLP head that produces class probabilities.
Why the Class Token Matters
- Global Information Aggregation: Through self-attention across all transformer layers, the CLS token attends to every patch in the image, gradually building a holistic representation of the entire visual scene.
- Task-Agnostic Representation: The CLS token learns a general-purpose image representation during pretraining that transfers effectively to diverse downstream tasks.
- Decoupled from Spatial Structure: Unlike CNN global average pooling, the CLS token is not tied to any spatial location — it can learn complex non-linear combinations of patch information through attention.
- Clean Architectural Separation: The CLS token cleanly separates the "understanding" function (transformer encoder) from the "decision" function (classification head) without requiring architectural modifications.
- BERT Compatibility: Using a CLS token maintains architectural consistency with NLP transformers, enabling shared research insights and multimodal fusion between vision and language models.
How the CLS Token Works
Layer 1 (Early):
- CLS token attends broadly to all patches with roughly uniform attention weights.
- Captures low-level global statistics (average color, overall brightness, texture distribution).
Middle Layers:
- Attention becomes more selective — CLS token focuses on informative patches (objects, distinctive features).
- Builds intermediate feature representations combining local and global context.
Final Layers:
- CLS token has attended to all patches across all layers through residual connections.
- Contains a rich, compressed representation of the entire image's semantic content.
Classification Head:
- The CLS token's final hidden state (768-D for ViT-Base) is passed through an MLP.
- MLP typically: Linear(768, num_classes) or Linear(768, hidden) → GELU → Linear(hidden, num_classes).
CLS Token vs. Global Average Pooling
| Aspect | CLS Token | Global Average Pooling (GAP) |
|---|---|---|
| Mechanism | Learned attention-based aggregation | Simple mean of all patch tokens |
| Learnable | Yes (additional parameters) | No (fixed operation) |
| Flexibility | Can weight patches differently | Equal weight to all patches |
| Performance | Slightly better with large-scale pretraining | Competitive or better with less data |
| DeiT Default | CLS token used | — |
| MAE/BEiT | Often use GAP instead | Preferred in self-supervised ViTs |
Variants and Extensions
- Register Tokens: Recent work (Darcet et al., 2023) adds additional learnable tokens beyond CLS to serve as "registers" that reduce attention artifacts in patch tokens.
- Multiple CLS Tokens: Some architectures use separate CLS tokens for different tasks or scales in multi-task learning.
- CLS-Free ViTs: Models like MAE (Masked Autoencoders) and DINOv2 often use global average pooling instead of a CLS token, achieving competitive or superior results.
- Distillation Token (DeiT): A second class-like token trained to match a teacher model's predictions, used alongside the standard CLS token.
The class token is the lens through which a Vision Transformer sees the whole image — by attending to every patch across every layer, this single learned vector distills an entire image into a representation rich enough to drive accurate classification and transfer learning.