Token Pruning

Token Pruning is an efficiency technique that removes uninformative tokens during inference — reducing the number of tokens processed by subsequent transformer layers, speeding up inference proportionally to the fraction of tokens removed.

How Does Token Pruning Work?

• Score: Compute an importance score for each token (CLS attention, gradient magnitude, learned predictor).
• Prune: Remove tokens with scores below a threshold (or keep top-$k$).
• Continue: Remaining tokens are processed by subsequent layers with reduced computation.
• Examples: DynamicViT, EViT, ToMe (Token Merging), A-ViT.

Why It Matters

• Inference Speed: 30-50% token reduction → proportional speedup with minimal accuracy loss (<0.5%).
• Adaptive: Different images have different amounts of informative content — pruning adapts automatically.
• Deployable: No architectural changes needed — can be applied to pre-trained models.

Token Pruning is throwing away what doesn't matter — dynamically removing uninformative tokens to accelerate transformer inference.