Phrase Masking

Phrase Masking is a masking strategy that masks complete phrases or multi-word expressions during pre-training — instead of masking individual tokens, entire meaningful phrases (noun phrases, verb phrases, prepositional phrases) are masked together, encouraging the model to learn phrase-level semantics.

Phrase Masking Approach

• Phrase Detection: Use constituency parsing, chunking, or n-gram frequency to identify meaningful phrases.
• Span Masking: Mask the entire phrase as a single unit — all tokens in the phrase are masked simultaneously.
• SpanBERT: Masks random contiguous spans (1-10 tokens) — a simplified form of phrase masking.
• ERNIE: Combines entity masking and phrase masking — multiple granularities of masking.

Why It Matters

• Compositional Semantics: Phrase masking forces the model to understand how words compose into meaningful units.
• Longer Dependencies: Predicting entire phrases requires understanding longer-range context than single tokens.
• Better Spans: Models trained with phrase masking perform better on span-level tasks — extractive QA, NER, coreference resolution.

Phrase Masking is hiding whole phrases — masking complete multi-word expressions to teach the model compositional phrase-level understanding.