Galactica
Galactica is a 120 billion parameter open-source language model trained on scientific and academic texts from arXiv, PubMed, mathematics repositories, and academic papers by Meta AI, designed specifically for scientific reasoning and knowledge retrieval—pioneering domain-specialized frontier-scale LLMs and exploring whether models trained on pure high-quality academic data outperform general internet-trained models on intellectual tasks.
Scientific Text Specialization
| Training Data | Quantity | Purpose |
|---|---|---|
| arXiv papers | Scientific preprints | Physics, ML, mathematics |
| PubMed | Biomedical literature | Medicine, biology research |
| Mathematics | Symbolic reasoning | Equation understanding |
| Academic papers | Peer-reviewed knowledge | Quality-filtered information |
Galactica was trained exclusively on high-quality-curated scientific and academic sources—a radical departure from web-scale models trained on noisy internet data.
Novel Capabilities: Galactica introduced scientific prompting:
- Citation generation (predicting relevant academic references)
- Equation understanding (recognizing when models misunderstand math)
- Table and figure interpretation from papers
Intended Purpose: Enable scientists to query scientific literature as natural language, enabling retrieval and reasoning across millions of papers—essentially making AI assistants for scientific research.
Reception & Lessons: Galactica was controversially released then quickly withdrawn when researchers documented concerning errors (hallucinating fake papers and citations). This taught the community valuable lessons about risk assessment for specialized models.
Legacy: Despite challenges, Galactica inspired the domain-specialized LLM trend. Models like Falcon (code-optimized, Stable Diffusion for text, etc.) followed the principle that training on curated domain data produces better specialist models.