Tokenization
Tokenization is the process of converting raw text into a sequence of discrete tokens (subword units) that serve as the input vocabulary for language models — determining how text is segmented into meaningful units, where the tokenizer's vocabulary size and algorithm directly impact model performance, multilingual capability, and inference efficiency.
Tokenization Approaches
| Method | Granularity | Vocabulary Size | Example: "unhappiness" |
|---|---|---|---|
| Word-level | Full words | 50K-500K | ["unhappiness"] |
| Character-level | Single chars | 26-256 | ["u","n","h","a","p","p","i","n","e","s","s"] |
| BPE (Subword) | Subword units | 32K-100K | ["un", "happiness"] |
| Byte-level BPE | Byte sequences | 50K-100K | ["un", "happ", "iness"] |
Byte Pair Encoding (BPE)
1. Start with character vocabulary + special end-of-word token.
2. Count all adjacent character pairs in training corpus.
3. Merge the most frequent pair into a new token.
4. Repeat steps 2-3 until desired vocabulary size reached.
- Example: "l o w" appears 5 times → merge to "lo w" → "low" appears 5 times → merge to single token "low".
- Rare words split into subwords; common words become single tokens.
- GPT-2/3/4 use byte-level BPE (operates on bytes, not Unicode characters → handles any text).
WordPiece (BERT)
- Similar to BPE but merges based on likelihood improvement, not frequency.
- Merge pair that maximizes: $\log P(AB) - \log P(A) - \log P(B)$.
- Uses ## prefix for continuation tokens: "playing" → ["play", "##ing"].
- Vocabulary: 30,522 tokens for BERT.
SentencePiece
- Language-agnostic: Treats input as raw Unicode bytes — no pre-tokenization (no word splitting rules).
- Supports BPE and Unigram methods.
- Unigram: Start with large vocab → iteratively remove tokens that least affect likelihood.
- Used by: T5, LLaMA, mBART, XLM-R.
- Advantage: Handles any language (CJK, Arabic, etc.) without language-specific rules.
Vocabulary Size Impact
| Vocab Size | Tokens/Word | Sequence Length | Compute |
|---|---|---|---|
| 4K | ~2.5 | Long sequences | High |
| 32K | ~1.3 | Medium | Medium |
| 100K | ~1.1 | Short | Lower |
| 256K | ~1.0 | Shortest | Lowest |
- Larger vocab → shorter sequences → faster inference, but larger embedding table.
- GPT-4: ~100K tokens. LLaMA: 32K. LLaMA-3: 128K.
Tokenization Challenges
- Number handling: "123456" might tokenize as ["123", "456"] → model doesn't understand mathematical relationship.
- Multilingual fairness: English words are often single tokens; other languages get split into many subwords → higher cost per concept.
- Whitespace sensitivity: Leading spaces, tabs, newlines affect tokenization in surprising ways.
Tokenization is the often-overlooked foundation that constrains everything a language model can do — a poorly designed tokenizer wastes model capacity on suboptimal text segmentation, while a well-designed one enables efficient multilingual processing and better numerical reasoning.