Large Language Model Pretraining
Large Language Model Pre-training is the foundation stage of LLM development where a Transformer-based model is trained on trillions of tokens of text data using the next-token prediction objective — learning general language understanding, reasoning, and knowledge representation that enables downstream instruction-following, question-answering, and code generation through subsequent fine-tuning stages.
Pre-training Objective:
- Next-Token Prediction (Causal LM): given a sequence of tokens [t₁, t₂, ..., t_n], predict t_{n+1} from the context [t₁, ..., t_n]; loss = cross-entropy between predicted distribution and actual next token; causal attention mask prevents looking ahead
- Masked Language Modeling (BERT-style): randomly mask 15% of tokens, predict the original tokens from context; produces bidirectional representations but not directly useful for generation; used by encoder-only models (BERT, RoBERTa)
- Prefix LM / Encoder-Decoder: encoder processes prefix bidirectionally, decoder generates continuation autoregressively; T5, UL2 use this approach; enables both understanding and generation but adds architectural complexity
- Scaling Insight: the next-token prediction objective, despite its simplicity, induces emergent capabilities (reasoning, arithmetic, translation, code generation) that were not explicitly trained — capabilities emerge with sufficient scale of data and parameters
Training Data Pipeline:
- Data Sources: web crawl (Common Crawl, ~200TB raw), books (BookCorpus, Pile), code (GitHub, StackOverflow), scientific papers (arXiv, PubMed), Wikipedia, conversations (Reddit), and curated instruction data
- Data Quality Filtering: deduplication (MinHash, exact n-gram), quality scoring (perplexity-based filtering with a smaller model), toxic content removal, PII scrubbing, URL/boilerplate removal; quality filtering typically discards 80-90% of raw web crawl
- Data Mixing: balanced mixture of domains; research suggests weighting high-quality sources (books, Wikipedia) disproportionately improves downstream performance; Llama training mix: ~80% web, ~5% code, ~5% Wikipedia, ~5% books, ~5% academic
- Tokenization: BPE (Byte-Pair Encoding) or SentencePiece with vocabulary sizes of 32K-128K tokens; larger vocabularies compress text better (fewer tokens per word) but increase embedding table size; multilingual tokenizers require larger vocabularies
Scaling Laws:
- Chinchilla Scaling: optimal compute allocation is roughly 20× more tokens than parameters (Hoffmann et al. 2022); a 70B parameter model should train on ~1.4T tokens for compute-optimal performance
- Compute Budget: training a 70B model on 2T tokens requires ~1.5×10²⁴ FLOPs; at 40% hardware utilization on 2000 H100 GPUs, this takes ~30 days; cost approximately $2-5M in cloud compute
- Predictable Scaling: validation loss scales as a power law with compute: L(C) = a·C^(-α) with α ≈ 0.05; enables reliable prediction of model performance before expensive training runs
- Emergent Abilities: certain capabilities (chain-of-thought reasoning, few-shot learning, multi-step arithmetic) appear suddenly above specific parameter/data thresholds; unpredictable from smaller-scale experiments
Training Infrastructure:
- Parallelism: 3D parallelism combining data parallel (gradient sync across replicas), tensor parallel (split layers across GPUs), and pipeline parallel (different layers on different GPUs); FSDP/ZeRO for memory-efficient data parallelism
- Mixed Precision: BF16 training with FP32 master weights; loss scaling for numerical stability; Tensor Cores provide 2× throughput for BF16/FP16 operations
- Checkpointing: save model state every 1000-5000 steps for failure recovery; training runs encounter hardware failures on average every few days at 1000+ GPU scale; efficient checkpoint/restart critical for completion
- Monitoring: loss curves, gradient norms, learning rate schedules, and downstream benchmark evaluation tracked continuously; loss spikes indicate data quality issues or numerical instability requiring intervention
LLM pre-training is the computationally intensive foundation that creates the raw intelligence of modern AI systems — the combination of the deceptively simple next-token prediction objective with massive scale produces models with emergent reasoning, knowledge, and language capabilities that define the frontier of artificial intelligence.