Training Cost
Training Cost refers to the total computational resources, time, energy, and financial expense required to train a machine learning model — for large language models this has grown from thousands of dollars (GPT-2 in 2019) to tens of millions of dollars (GPT-4 in 2023) to projected hundreds of millions (frontier models in 2025+), driven by scaling laws that show model quality improves predictably with more compute, creating a compute arms race that makes training cost the defining constraint of modern AI development.
What Is Training Cost?
- Definition: The total expense of computing all the gradient updates needed to train a model to convergence — encompassing GPU/TPU rental or ownership, electricity, networking infrastructure, cooling, engineering salaries, data acquisition, and failed experiments.
- Why It Matters: Training cost determines who can build frontier AI models. When training costs reach $100M+, only a handful of organizations (OpenAI, Google, Meta, Anthropic, xAI) can compete. This has profound implications for AI concentration, accessibility, and safety.
- The Scaling Reality: Every 10× increase in training compute has historically delivered meaningful capability improvements, incentivizing ever-larger training runs.
Training Cost of Notable Models
| Model | Year | Parameters | Training Compute | Estimated Cost | Hardware |
|---|---|---|---|---|---|
| GPT-2 | 2019 | 1.5B | ~1 PF-day | ~$50K | TPU v3 |
| GPT-3 | 2020 | 175B | ~3,640 PF-days | ~$4.6M | V100 cluster |
| PaLM | 2022 | 540B | ~25,000 PF-days | ~$8-12M | TPU v4 |
| LLaMA-2 70B | 2023 | 70B | ~6,000 PF-days | ~$2-4M | A100 cluster |
| GPT-4 | 2023 | ~1.8T (rumored) | ~100,000+ PF-days | ~$60-100M | A100 cluster |
| Llama 3 405B | 2024 | 405B | ~40,000 PF-days | ~$50-80M | H100 cluster |
| Frontier models | 2025+ | 1T+ | 500,000+ PF-days | ~$200-500M | H100/B200 clusters |
Components of Training Cost
| Component | Share of Total | Description |
|---|---|---|
| GPU/TPU Compute | 60-80% | Accelerator rental or amortized purchase cost |
| Electricity | 5-15% | Power for compute + cooling (training Llama-3: ~30 GWh) |
| Networking | 5-10% | InfiniBand/NVLink for distributed training communication |
| Engineering | 5-15% | ML researchers, systems engineers ($200-500K/year each) |
| Data | 2-5% | Acquisition, cleaning, filtering, human annotation |
| Failed Experiments | 20-50% of total budget | Hyperparameter searches, diverged runs, restarts |
Cost Optimization Strategies
| Strategy | Savings | Trade-off |
|---|---|---|
| Mixed Precision (FP16/BF16) | ~2× throughput | Negligible quality loss with loss scaling |
| Gradient Checkpointing | ~60% memory reduction | 20-30% slower (recomputation) |
| Data Parallelism | Near-linear scaling to 1000s of GPUs | Communication overhead at extreme scale |
| MoE Architecture | 3-5× less compute per token for same quality | Higher total memory, routing complexity |
| Efficient Architectures (FlashAttention) | 2-3× attention speedup | Minor implementation effort |
| Spot/Preemptible Instances | 60-70% cost reduction | Requires checkpointing, interruption handling |
| Distillation | Train small model from large model outputs | Requires teacher model (already trained) |
Training Cost is the defining constraint of modern AI development — scaling from thousands to hundreds of millions of dollars as models grow in size and capability, determining which organizations can build frontier AI systems, driving the development of cost-reduction techniques from mixed precision to MoE architectures, and raising fundamental questions about the concentration, sustainability, and accessibility of advanced AI research.