Gradient-Based Masking

Gradient-Based Masking is a technique that selects tokens to mask based on their influence on the loss gradient — identifying tokens that are most critical for the model's current state or that provide the strongest training signal.

Mechanism

• Saliency: Compute gradients with respect to input tokens. High gradient = this token matters a lot.
• Selection: Mask tokens with high gradients (force the model to find alternative paths to meaning) OR mask tokens that maximize expected loss.
• One-Shot: Requires a backward pass to find masks, then another pass to train — computationally expensive (2x cost).

Why It Matters

• Adversarial: Acts like adversarial training — attacking the model's reliance on specific keywords.
• Interpretability: Reveals which tokens the model relies on.
• Cost: Usually too expensive for large-scale pre-training compared to random dynamic masking.

Gradient-Based Masking is mathematically targeted hiding — using the model's own internal gradients to decide which words are most important to hide.