Dynamic Masking

Dynamic Masking is a training strategy for Masked Language Models (like RoBERTa) where the mask pattern is generated on-the-fly every time a sequence is fed to the model, rather than being generated once and saved (Static Masking) — allowing the model to see different versions of the same sentence with different masks over training epochs.

Dynamic vs. Static

• Static (Original BERT): Data was masked once during preprocessing. The model saw the exact same mask pattern for "Sentence A" in Epoch 1, 2, 10.
• Dynamic (RoBERTa): Mask is applied in the data loader. Epoch 1: "The [MASK] brown...", Epoch 2: "The quick [MASK]...".
• Benefit: Effectively multiplies the dataset size — the model never "memorizes" the specific mask solution.

Why It Matters

• Performance: RoBERTa showed that dynamic masking improves performance significantly over static masking.
• Epochs: Allows training for more epochs without overfitting to specific masks.
• Standard Practice: Now standard in almost all MLM training pipelines.

Dynamic Masking is reshuffling the problem — changing which words are hidden every time the model studies a sentence to prevent memorization.