Llemma
Llemma is a 34-billion parameter open-source mathematics language model fine-tuned from Code Llama on mathematical texts, competition problems, and formal proofs, representing the first open-source model demonstrating frontier mathematical reasoning and proof-retrieval capability on university-level mathematics at a scale matching proprietary systems like GPT-4.
Code + Math Fusion
Llemma combines two fundamental insights:
| Foundation | Source | Benefit |
|---|---|---|
| Code Llama 34B | Meta AI's code specialist | Code understanding improves math (symbolic manipulation) |
| Mathematical Data | arXiv, MATH dataset, proofs | Domain-specific reasoning enhancement |
Llemma fine-tunes the already code-competent Code Llama on mathematical texts and formal proofs—recognizing that mathematics is symbolic computation similar to programming.
Proof Retrieval & Generation: Unique capability to retrieve and generate formal mathematical proofs—not just answers but rigorous derivations. This bridges neural LLMs (pattern matching) with symbolic mathematics (rigorous reasoning).
Performance: Achieves 47.3% on MATH (university-level competition problems)—competitive with GPT-3.5 and matching proprietary systems. First fully open model at this level.
Tools Integration: Designed to pair with symbolic math tools (SageMath, Mathematica)—enabling hybrid workflow where LLM handles reasoning and symbolic systems provide verification.
Legacy: Proves that open-source mathematics specialists can reach frontier capability—democratizing access to advanced mathematical reasoning and enabling researchers to study how LLMs understand formal proofs.