Deberta

DeBERTa (Decoding-enhanced BERT with Disentangled Attention) is a pre-trained language model that improves upon BERT by disentangling content and position representations — computing separate attention for content-to-content, content-to-position, and position-to-content interactions.

Key Innovations of DeBERTa

• Disentangled Attention: Separate matrices for content (word) and position, with three attention components instead of one.
• Enhanced Mask Decoder (EMD): Uses absolute position information in the decoder layer for MLM prediction.
• Virtual Adversarial Training: Fine-tuning with perturbation-based regularization.
• Paper: He et al. (2021, Microsoft).

Why It Matters

• SuperGLUE #1: First model to surpass human baseline on the SuperGLUE benchmark.
• Disentanglement: Separating content and position allows the model to learn cleaner representations.
• DeBERTaV3: Subsequent versions with ELECTRA-style training further improved efficiency.

DeBERTa is BERT with separated content and position — disentangling what a word means from where it appears for more powerful language understanding.