Entity Masking

Entity Masking is a masking strategy that preferentially masks named entities (people, organizations, locations, dates) during pre-training — targeting semantically important spans rather than random tokens, forcing the model to learn world knowledge and entity-level understanding.

Entity Masking Approach

• Entity Detection: Use NER (Named Entity Recognition) to identify entities in the training text.
• Preferential Masking: Mask entire entities more frequently than random tokens — focus learning on factual knowledge.
• Entity Types: Person names, organization names, locations, dates, quantities — semantically meaningful spans.
• ERNIE: Baidu's ERNIE (Enhanced Representation through Knowledge Integration) popularized entity and phrase masking.

Why It Matters

• Knowledge Acquisition: Entity masking forces the model to memorize and reason about real-world entities — better knowledge representation.
• Downstream Tasks: Improves performance on knowledge-intensive tasks — question answering, relation extraction, entity typing.
• Knowledge Graphs: Can be combined with knowledge graph embeddings for enhanced entity understanding.

Entity Masking is hiding the important names — forcing the language model to learn world knowledge by preferentially masking named entities during pre-training.