How AI Tokenization Works: Byte-Pair Encoding (BPE) vs SentencePiece
Why can't neural networks just read raw letters? How does a 200,000-word vocabulary compress multilingual documents? Here is the exact mathematical and algorithmic reality of modern LLM tokenizers.
1. The Problem With Character-Level and Word-Level Processing
Neural language models do not process text strings directly; they compute over vectors of floating-point embeddings indexed by discrete integers. When computer scientists first designed text models, two naive approaches failed:
- Word-level tokenization: Assigning an ID to every unique English word creates an intractable vocabulary (millions of entries), cannot handle typos (
tehinstead ofthe), and fails completely on out-of-vocabulary (OOV) terms. - Character-level tokenization: Using single UTF-8 characters keeps the vocabulary tiny (256 bytes), but blows up sequence length by 4x to 5x. Because transformer self-attention scales quadratically or linearly with sequence length, processing character-by-character is prohibitively expensive.
2. How Byte-Pair Encoding (BPE) Solves the Dilemma
Originally a data compression algorithm published by Philip Gage in 1994, Byte-Pair Encoding was adapted for neural networks by Sennrich et al. in 2016.
BPE begins with a base vocabulary of individual bytes (256 items). It iteratively scans a multi-terabyte training corpus, finds the most frequently co-occurring pair of adjacent tokens, merges them into a new single token, and repeats this process until reaching a predetermined vocabulary budget (e.g. 100,000 or 200,000 tokens).
3. Frontier Tokenizer Architectures Compared
Today, different frontier AI labs utilize specialized tokenizer implementations tailored to their training data and target languages:
| Model Family | Tokenizer Architecture | Vocabulary | Key Characteristics |
|---|---|---|---|
| OpenAI GPT-6 / o3 | o200k_base (Tiktoken) | 200,000 | High compression on non-English scripts; fewer tokens per word. |
| Anthropic Claude 5 | Claude BPE | ~65,000 | Optimized for source code syntax and technical reasoning. |
| Google Gemini 3 | SentencePiece Unigram | 256,000 | Massive vocabulary with native whitespace treatment. |
4. Why Tokenization Matters for Your API Bill
Because AI providers charge per 1M tokens rather than per word or character, tokenizer efficiency directly dictates cost. If Model A uses 1.1 tokens per word on your multilingual codebase while Model B requires 1.8 tokens per word, Model B effectively charges a 63% hidden premium even if their posted per-token rates appear identical.