Whole Word Masking

Whole Word Masking is a masking strategy for masked language model pre-training where entire words are masked rather than individual subword tokens — when a word is selected for masking, ALL its constituent subword pieces (WordPiece, BPE) are masked together, preventing the model from trivially predicting a masked piece from its sibling pieces.

Whole Word Masking Details

• Standard Masking: BERT randomly masks individual tokens — a word like "playing" tokenized as "play" "##ing" might only mask "##ing."
• Problem: If only "##ing" is masked, the model can trivially predict it from "play" — learns subword patterns, not semantics.
• WWM: If "playing" is selected, BOTH "play" AND "##ing" are masked — forces the model to use context.
• Rate: Still masks ~15% of the original words — but the token masking rate may differ.

Why It Matters

• Better Representations: WWM produces better contextualized representations — the model must learn deeper semantic understanding.
• BERT-WWM: Google released Whole Word Masking variants of BERT — improved performance on downstream tasks.
• Chinese: Particularly important for Chinese BERT — Chinese word segmentation makes token-level masking problematic.

Whole Word Masking is masking complete words, not fragments — forcing the model to learn from context rather than subword co-occurrence patterns.