Tokenization

Tokenization splits text into tokens (subwords, characters, or words) for model input processing. Why subword tokenization?: Balance vocabulary size vs sequence length. Rare words split into known pieces. Handles OOV (out-of-vocabulary) naturally. Major algorithms: BPE (Byte-Pair Encoding): Iteratively merge most frequent character pairs, used by GPT models. WordPiece: Similar to BPE, used by BERT. SentencePiece: Language-agnostic, includes BPE and Unigram variants. Unigram: Starts with large vocabulary, prunes based on likelihood. Key trade-offs: Smaller vocabulary means more tokens per text means longer sequences. Larger vocabulary means shorter sequences but more embeddings to learn. Token characteristics: Average 4 characters/token in English for GPT. Spaces often attached to following token. Non-English may use more tokens. Impact on cost/limits: API pricing often per-token, context limits in tokens, tokenization efficiency affects multilingual performance. Special tokens: CLS, SEP, PAD, BOS, EOS for model-specific purposes. Tools: tiktoken (OpenAI), tokenizers (HuggingFace). Understanding tokenization essential for cost estimation and prompt engineering.