Domain-Adaptive Pre-Training

Domain-Adaptive Pre-training (DAPT) is the process of taking a general-purpose pre-trained model (like BERT) and continuing to pre-train it on a large corpus of unlabeled text from a specific domain (e.g., biomedical, legal, financial) — adapting the model's vocabulary and statistical understanding to the target domain before fine-tuning.

Process (Don't Stop Pre-training)

• Source: Start with RoBERTa (trained on CommonCrawl).
• Target: Continue training MLM on all available Biomedical papers (PubMed).
• Result: "BioRoBERTa" — better at medical jargon and scientific reasoning.
• Fine-tune: Finally, fine-tune on the specific medical task (e.g., diagnosis prediction).

Why It Matters

• Vocabulary Shift: "Virus" means something different in biology vs. computer security. DAPT updates context.
• Performance: Significant gains on in-domain tasks compared to generic models.
• Cost: Much cheaper than pre-training from scratch on domain data.

Domain-Adaptive Pre-training is specializing the expert — sending a generalist model to law school or med school to learn the specific language of a field.