Dataset

Training Data for LLMs

Pretraining Datasets
Large language models are pretrained on massive text corpora—often trillions of tokens from diverse sources.

Common Pretraining Sources
| Source | Content | Scale |
|--------|---------|-------|
| Common Crawl | Web pages | Petabytes |
| The Pile | Curated diverse text | 825 GB |
| Wikipedia | Encyclopedia articles | ~20 GB |
| Books3 | Books | ~100 GB |
| GitHub | Source code | ~150 GB |
| ArXiv | Scientific papers | ~90 GB |
| Stack Exchange | Q&A | ~60 GB |

Data Processing Pipeline
1. Crawling: Collect raw text from sources
2. Deduplication: Remove duplicate documents
3. Filtering: Remove low-quality, toxic, or harmful content
4. Language detection: Filter by language if needed
5. Tokenization: Convert to token sequences
6. Shuffling: Randomize for training

Fine-Tuning Datasets

By Task Type
| Task | Datasets | Size |
|------|----------|------|
| Instruction | Alpaca, Dolly, OpenAssistant | 15K-200K |
| Code | CodeAlpaca, StarCoder data | 20K-1M |
| Math | GSM8K, MATH | 8K-12K |
| Dialogue | ShareGPT, UltraChat | 50K-1M |
| Safety | Anthropic HH-RLHF | 160K |

Data Quality Principles

Quality > Quantity
Research shows that smaller, high-quality datasets often outperform larger noisy ones:
- Phi-1: 1.3B model trained on 6B tokens of textbook-quality data
- LIMA: 1K carefully curated examples for instruction tuning

Key Quality Factors
- Accuracy: Factually correct information
- Diversity: Wide coverage of topics and styles
- Consistency: Uniform formatting and quality standards
- Recency: Up-to-date information when relevant
- Safety: No harmful, biased, or toxic content

Legal Considerations
- Respect copyright and licensing
- Consider opt-out mechanisms for data subjects
- Document data provenance for compliance