Vocabulary Size Selection
Vocabulary size selection is the decision process for choosing tokenizer vocabulary cardinality to balance compression, coverage, and model efficiency - size choice strongly influences sequence length and memory behavior.
What Is Vocabulary size selection?
- Definition: Setting the number of learned token entries in the tokenizer vocabulary.
- Tradeoff Axis: Larger vocabularies shorten sequences but increase embedding parameters.
- Coverage Effect: Smaller vocabularies increase subword fragmentation for rare terms.
- Context Impact: Token granularity affects effective use of fixed context windows.
Why Vocabulary size selection Matters
- Compute Planning: Vocabulary size changes embedding memory and inference throughput.
- Domain Performance: Technical corpora may need enough tokens for frequent specialized terms.
- Training Efficiency: Overly fragmented tokenization can slow convergence.
- Serving Cost: Sequence length directly affects attention compute in transformer models.
- Quality Stability: Balanced sizes reduce both OOV fragmentation and parameter bloat.
How It Is Used in Practice
- Curve Analysis: Plot sequence-length reduction versus vocabulary growth on target corpora.
- Task Benchmarks: Compare downstream quality across candidate vocabulary sizes.
- Lifecycle Review: Reassess size decisions when language mix or product domain shifts.
Vocabulary size selection is a central tokenizer design decision with system-wide impact - correct sizing improves model efficiency and domain coverage simultaneously.