Tokenizers
Hugging Face Tokenizers is a high-performance text tokenization library written in Rust with Python bindings that converts raw text into the token sequences that language models consume — implementing BPE (Byte-Pair Encoding), WordPiece, Unigram, and SentencePiece algorithms at speeds of 1 GB of text in under 20 seconds, with character-level alignment tracking that maps every token back to its exact position in the original string.
What Is Tokenizers?
- Definition: A Rust-based library that handles the critical first step of any NLP pipeline — splitting raw text into subword tokens (pieces of words) that map to the vocabulary entries a language model was trained on, then converting those tokens to integer IDs for model input.
- Rust Performance: Written in Rust with Python bindings via PyO3 — tokenizing 1 GB of text takes under 20 seconds, compared to minutes or hours with pure Python implementations. The speed difference matters for preprocessing large training datasets.
- Alignment Tracking: Tokenizers maintains a character-to-token offset mapping — for every token, you can query exactly which characters in the original string it corresponds to. This is critical for NER tasks where you need to map token-level predictions back to character spans ("highlight the entity in the original text").
- Parallelism: Automatic multi-core parallelism for batch tokenization — tokenizing thousands of documents uses all available CPU cores without explicit threading code.
- Training: Train custom tokenizers from scratch on your own corpus — `tokenizer.train(files, trainer)` learns a vocabulary optimized for your domain (medical text, code, legal documents).
Tokenization Algorithms
| Algorithm | Used By | How It Works |
|---|---|---|
| BPE (Byte-Pair Encoding) | GPT-2, GPT-4, LLaMA, Mistral | Iteratively merges most frequent character pairs |
| WordPiece | BERT, DistilBERT, Electra | Greedy longest-match from vocabulary |
| Unigram | T5, ALBERT, XLNet | Probabilistic subword selection |
| SentencePiece | LLaMA, T5, mBART | Language-agnostic BPE/Unigram on raw text |
Key Features
- Pre-tokenization: Configurable text splitting before subword tokenization — whitespace splitting, punctuation splitting, byte-level BPE, or custom regex patterns.
- Normalization: Unicode normalization (NFC, NFKC), lowercasing, accent stripping — applied consistently before tokenization.
- Post-processing: Automatic addition of special tokens ([CLS], [SEP], <s>, </s>) — configured per model architecture.
- Truncation and Padding: Built-in truncation to max length and padding to batch length — with multiple truncation strategies (longest_first, only_first, only_second).
- Fast Tokenizers in Transformers: `AutoTokenizer.from_pretrained()` automatically loads the Rust-backed fast tokenizer when available — providing the same API as the Python tokenizer with 10-100× speed improvement.
Hugging Face Tokenizers is the high-performance foundation that every Transformers model depends on — converting raw text to model-ready token sequences at Rust speed with character-level alignment tracking, making it both the invisible workhorse of the Hugging Face ecosystem and an essential tool for teams training custom tokenizers on domain-specific corpora.