Enhanced Mask Decoder

Enhanced Mask Decoder (EMD) is a component of DeBERTa that incorporates absolute position information in the final decoding layer — compensating for the fact that disentangled attention uses only relative positions, which is insufficient for tasks like masked language modeling.

How Does EMD Work?

• Problem: Relative position alone cannot distinguish "A new [MASK] opened" → "store" vs "A new store [MASK]" → "opened". Absolute position matters.
• Solution: Add absolute position embeddings only in the final decoder layer before the MLM prediction head.
• Minimal Disruption: Most layers use relative position (better generalization). Only the decoder uses absolute position (for disambiguation).

Why It Matters

• Position Disambiguation: Absolute position is necessary for predicting masked tokens correctly in certain contexts.
• Best of Both: Combines relative position (better generalization) with absolute position (necessary disambiguation).
• DeBERTa Architecture: EMD is the third key innovation of DeBERTa alongside disentangled attention and virtual adversarial training.

EMD is the final position anchor — adding absolute position information at the last moment so the model knows exactly where each prediction should go.