Vocabulary Size

Vocabulary size defines the number of unique tokens a model can represent, balancing coverage against efficiency. Trade-offs: Smaller vocabulary (8K-32K): Longer sequences (more tokens per text), smaller embedding matrix, harder time with rare words. Larger vocabulary (50K-150K): Shorter sequences, larger embedding matrix, better coverage of subwords. Typical sizes: GPT-2: 50,257. LLaMA: 32,000. BERT: 30,522. GPT-4: around 100K. Impact on training: Vocabulary size affects embedding parameter count (vocab_size x hidden_dim), softmax computation cost, memory usage. Impact on inference: Affects token usage (API costs), context window utilization, per-token information density. Language considerations: English-centric vocabularies inefficient for other languages. Multilingual models need larger vocabularies. Specialized vocabularies: Code models may add programming tokens, domain models may add technical terms. Hyperparameter choice: Balance between vocabulary size, sequence length, and model capacity. No universally optimal size.