Dynamic Token Pruning

Dynamic Token Pruning is a token pruning approach where the pruning decisions are made dynamically at each layer based on learned criteria — allowing different layers to prune different tokens, and different inputs to have different pruning patterns.

How Does Dynamic Token Pruning Work?

• Per-Layer Decision: At each layer, a lightweight predictor determines which tokens to keep.
• Progressive: Early layers may keep most tokens; later layers prune more aggressively.
• Learned Pruning: The pruning predictor is trained jointly with the main network (Gumbel-softmax or straight-through estimator).
• Example: DynamicViT uses a prediction module trained with a distillation loss.

Why It Matters

• Input-Adaptive: Easy images prune many tokens early. Complex images retain more tokens longer.
• Layer-Adaptive: Different layers can focus on different tokens — earlier layers keep diverse tokens, later layers keep only task-relevant ones.
• Accuracy: Trained pruning predictors maintain accuracy better than heuristic pruning methods.

Dynamic Token Pruning is learned selective attention — training the model to automatically decide which tokens to keep at each layer for optimal efficiency.