Adaptive Masking

Adaptive Masking refers to strategies where the masking rate or pattern changes during training based on the model's performance or curriculum — focusing learning on "hard" tokens or adjusting difficulty as the model improves.

Approaches

• Hardness-based: Mask tokens the model is currently predicting *well* (too easy) or *poorly* (need focus)? Typically, masking "hard" or "salient" tokens is better.
• Rate Scheduling: Start with low masking rate (easy), increase to high masking rate (hard).
• Model-based: Use a smaller model to identify "important" tokens to mask for a larger model.

Why It Matters

• Efficiency: Don't waste compute predicting "the", "a", "is" (easy stop words).
• Learning: Force the model to solve difficult semantic relations.
• Complexity: Adds complexity to the training pipeline — simple random masking is often "good enough" and surprisingly hard to beat.

Adaptive Masking is smart masking — changing *what* or *how much* to hide based on what the model already knows.