Goat
GOAT (Good at Arithmetic Tasks) is a Llama-based language model fine-tuned specifically for arithmetic calculation, demonstrating that targeted synthetic data training can solve the fundamental tokenization problem that makes standard LLMs fail at basic math — achieving state-of-the-art performance on multi-digit addition, subtraction, multiplication, and division by training on carefully structured arithmetic examples that teach the model columnar computation strategies, even outperforming GPT-4 on certain large-number operations at time of release.
Why LLMs Fail at Arithmetic
- Tokenization Problem: Standard LLMs tokenize "12345" as subword chunks like "123" + "45" or "1" + "2345" — destroying the digit-level alignment needed for columnar arithmetic. The model literally cannot see individual digits in consistent positions.
- Pattern vs. Computation: LLMs learn statistical patterns, not algorithms. They memorize that "2+2=4" from training data but cannot generalize to "47293+81956" because that specific sum was never in training.
- Carry Propagation: Multi-digit addition requires carrying across columns — a sequential, algorithmic process that autoregressive generation handles poorly without explicit training.
The GOAT Solution
| Component | Approach | Result |
|---|---|---|
| Base Model | Llama-7B | Strong language understanding foundation |
| Training Data | Synthetic arithmetic dataset with step-by-step solutions | Teaches columnar computation |
| Format | "Q: 47293 + 81956 = ? A: Let me compute step by step..." | Chain-of-thought arithmetic |
| Operations | Addition, subtraction, multiplication, division | Full arithmetic coverage |
Key Innovation: GOAT's training data presents arithmetic problems with explicit intermediate steps — showing the model how to align digits, propagate carries, and verify results. This transforms arithmetic from pattern-matching into learned algorithmic execution.
Performance
| Task | GOAT-7B | GPT-4 | Llama-7B (base) |
|---|---|---|---|
| Large addition (10+ digits) | 99%+ | ~85% | <10% |
| Large multiplication | 95%+ | ~70% | <5% |
| Division with remainders | 90%+ | ~80% | <5% |
Significance: GOAT proved that domain-specific fine-tuning on synthetic data can solve fundamental LLM limitations — the tokenization problem isn't inherent to the architecture but addressable through targeted training. This influenced subsequent math-specialized models (MAmmoTH, MetaMath, Llemma) and validated the approach of using synthetic datasets to teach LLMs algorithmic reasoning.
GOAT is a landmark demonstration that LLMs can learn genuine computation — proving that fine-tuning with structured arithmetic examples enables models to perform reliable multi-digit calculation that base models and even frontier systems struggle with, establishing synthetic data as the key to teaching algorithmic skills.