Tinyllama
TinyLlama is a 1.1 billion parameter language model trained on 3 trillion tokens, demonstrating that scaling laws apply even at the smallest feasible scale and creating the largest openly available sub-2B model — proving that models under 10GB can achieve surprising logical capabilities when trained on enormous token counts, enabling practical deployment on mobile phones, retro hardware, and ultra-low-power devices while maintaining instruction-following and reasoning competence.
Revolutionary Training Scale
| Aspect | TinyLlama | Traditional Scaling |
|---|---|---|
| Parameters | 1.1B | Typically 7B+ considered viable minimum |
| Training Tokens | 3 trillion | Usually 300B-1.4T for small models |
| Token/Param Ratio | 2700x | Aggressive overtraining for size |
| Result | Surprising reasoning ability | Size constraints limiting capability |
Design Philosophy: Rather than normal token budgets, TinyLlama uses extreme overtraining (3 trillion tokens for 1.1B parameters) to compensate for limited parameter capacity—proving scaling laws hold across extreme ranges.
Accessibility: At 1.1B parameters, TinyLlama fits in under 2GB of memory. It runs on resourced-constrained devices: iPhones, Raspberry Pi, edge inferenceclusters, and retro computers—democratizing AI access beyond cloud-dependent systems.
Legacy: Established that scale and capability are separable concerns—smarter training (more tokens) can compensate for smaller models, enabling practical AI everywhere.