Tokenization: BPE & SentencePiece

Imagine trying to teach a computer to read. You can't just feed it raw text, because it doesn't know what letters, words, or sentences are. You need to break the text into small, meaningful pieces that the computer can work with. This process is called tokenization, and it's the very first step in how every language model processes your input.

Think of it like breaking a sentence into Scrabble tiles. You need pieces that are small enough to be flexible, but large enough to carry meaning. The word "unhappiness" might become ["un", "happi", "ness"]. Each piece is meaningful on its own, and together they reconstruct the whole word. Every large language model (LLM) sees the world through these subword tokens, not raw words or characters.

This article is your foundation. Every topic that follows, like attention, embeddings, and inference, builds on understanding how raw text becomes the numerical sequences models actually process.

💡 Key insight: Tokenization is the bridge between human language and model computation. Understanding how and why text gets split into tokens will help you reason about everything from API costs to why some languages perform worse than others in AI systems.

---

The core problem: why not just use words or characters?

Before looking at the algorithms, it's important to understand the core problem:

Subword tokenization solves this by learning a vocabulary of frequently co-occurring character sequences. Common words like "the" stay whole. Rare words like "unhappiness" get split into meaningful pieces: ["un", "happi", "ness"]. This is the approach used by every modern LLM.

---

Byte pair encoding (BPE)

BPE is the dominant tokenization algorithm in modern LLMs. Originally a data compression algorithm invented by Philip Gage in 1994^[1], it was adapted for Natural Language Processing (NLP) by Sennrich, Haddow & Birch (2016)^[2] and later refined into byte-level BPE by Radford et al. (2019)^[3] for GPT-2.

How BPE training works

The algorithm is simple: it iteratively merges the most frequent pair.

Here's the complete training algorithm. It takes a corpus of text and a target vocabulary size as input, and outputs the final vocabulary along with the ordered list of merge rules that will be used to tokenize new text:

python
1def merge_pair(ids: list, pair: tuple, new_id: int) -> list:
2    """Replaces all occurrences of a pair with a new_id in a list of ids."""
3    new_ids = []
4    i = 0
5    while i < len(ids):
6        if i + 1 < len(ids) and (ids[i], ids[i+1]) == pair:
7            new_ids.append(new_id)
8            i += 2
9        else:
10            new_ids.append(ids[i])
11            i += 1
12    return new_ids
13
14def train_bpe(corpus: str, vocab_size: int) -> tuple[dict, dict]:
15    """Train a BPE tokenizer from a text corpus."""
16    
17    # Step 1: Encode text as UTF-8 bytes (byte-level BPE)
18    # This gives us a base vocabulary of exactly 256 byte values
19    token_ids = list(corpus.encode("utf-8"))  # e.g., [72, 101, 108, ...]
20    
21    merges = {}  # (pair) -> new_token_id
22    vocab = {i: bytes([i]) for i in range(256)}  # Base: all byte values
23    
24    num_merges = vocab_size - 256
25    
26    for i in range(num_merges):
27        # Step 2: Count all adjacent pairs
28        pair_counts = {}
29        for j in range(len(token_ids) - 1):
30            pair = (token_ids[j], token_ids[j + 1])
31            pair_counts[pair] = pair_counts.get(pair, 0) + 1
32        
33        if not pair_counts:
34            break
35        
36        # Step 3: Find the most frequent pair
37        best_pair = max(pair_counts, key=pair_counts.get)
38        
39        # Step 4: Create new token and replace all occurrences
40        new_id = 256 + i
41        token_ids = merge_pair(token_ids, best_pair, new_id)
42        
43        merges[best_pair] = new_id
44        vocab[new_id] = vocab[best_pair[0]] + vocab[best_pair[1]]
45    
46    return vocab, merges  # Ordered merge rules are the "model"

Concrete example: BPE training step by step

Let's trace BPE on a tiny corpus to see how it works. We input a simple repeating phrase, and we observe how the algorithm incrementally merges characters into longer subwords:

text
1Corpus: "low low low low lowest lowest newer newer newer wider wider"
2
3Step 0: Initial tokens (characters):
4  ['l','o','w',' ','l','o','w',' ','l','o','w',' ','l','o','w',' ',
5   'l','o','w','e','s','t',' ','l','o','w','e','s','t',' ',
6   'n','e','w','e','r',' ','n','e','w','e','r',' ','n','e','w','e','r',' ',
7   'w','i','d','e','r',' ','w','i','d','e','r']
8
9Step 1: Most frequent pair: ('l','o') appears 6 times
10  Merge: 'l' + 'o' → 'lo'
11
12Step 2: Most frequent pair: ('lo','w') appears 6 times
13  Merge: 'lo' + 'w' → 'low'
14
15Step 3: Most frequent pair: ('e','r') appears 5 times
16  Merge: 'e' + 'r' → 'er'
17
18Step 4: Most frequent pair: ('e','s') appears 2 times
19  Merge: 'e' + 's' → 'es'
20
21Step 5: Most frequent pair: ('es','t') appears 2 times
22  Merge: 'es' + 't' → 'est'
23
24Final vocabulary includes: {..., 'lo', 'low', 'er', 'es', 'est', ...}

💡 Key insight: The merge rules capture linguistic patterns. "est" emerges as a suffix, "er" as another. BPE discovers morphology without being told about it.

Tokenization at inference time

Once trained, the merge rules are applied in order to new text. This process takes a raw string, splits it into characters (or bytes in modern implementations), and sequentially applies the learned merges to output the final subword tokens and their IDs:

The following step-by-step example demonstrates this inference process. We input the raw word "lowest" and systematically apply the sequence of learned merge rules to output the final subword tokens (low and est) and their corresponding IDs:

text
1Input: "lowest"
2
3Step 0: ['l', 'o', 'w', 'e', 's', 't']       ← start with characters
4Merge 1 ('l','o' → 'lo'):  ['lo', 'w', 'e', 's', 't']
5Merge 2 ('lo','w' → 'low'): ['low', 'e', 's', 't']
6Merge 3 ('e','s' → 'es'):   ['low', 'es', 't']
7Merge 4 ('es','t' → 'est'): ['low', 'est']
8
9Final tokens: ['low', 'est'] → IDs: [vocab['low'], vocab['est']]

Note: This example uses character-level BPE for clarity. Real implementations like GPT-2 use byte-level BPE, where the base vocabulary starts at 256 byte values instead of individual characters. The merge logic is identical; only the base token unit changes.

Byte-level BPE: the modern standard

GPT-2 introduced a critical innovation: byte-level BPE^[3]. Instead of starting from Unicode characters, it starts from raw UTF-8 bytes (a base vocabulary of exactly 256). This guarantees:

• •No unknown tokens: Every possible input can be encoded (since any text is a sequence of bytes).
• •Fixed base vocabulary: Always starts at 256, regardless of language.
• •Script agnostic: Handles Chinese, Arabic, emoji, and anything else that's valid UTF-8.

⚠️ Warning: Non-ASCII characters (like Chinese characters or emoji) require 2–4 bytes each, so they initially get split into multiple tokens before merges can compress them. This is one root cause of the multilingual "tokenization tax" (discussed later).

Adoption in the industry

Because of these advantages, byte-level BPE has become the industry standard for generative models. GPT-2, OpenAI's tiktoken-based models, Qwen, Mistral, Code Llama, and StarCoder all rely on it. Virtually every modern autoregressive LLM uses BPE.

---

WordPiece

WordPiece was developed by Schuster & Nakajima (2012)^[4] at Google for Japanese and Korean voice search, then popularized by BERT (Devlin et al., 2019)^[5].

How WordPiece differs from BPE

The key difference is that WordPiece merges are chosen to maximize the likelihood of the training corpus, not just the raw frequency. The merge score is:

$\text{score}(a, b) = \frac{\text{freq}(ab)}{\text{freq}(a) \times \text{freq}(b)}$

Reading the formula

This formula divides the frequency of the merged pair by the product of the individual frequencies. This is essentially pointwise mutual information (PMI), which measures how much more likely two tokens appear together than you'd expect by chance. A pair like ("q", "u") scores high because "qu" is far more common than the individual frequencies of "q" and "u" would predict. Unlike BPE (which just picks the most frequent pair), WordPiece picks the pair with the strongest statistical association.

BPE vs WordPiece: the merge decision

Let's look at a scenario that shows the difference. Given the following token frequencies, BPE chooses the most frequent pair, while WordPiece chooses the one with the highest statistical association:

text
1Token frequencies: "t" appears 1000x, "h" appears 800x, "th" appears 500x
2                   "x" appears 10x,  "z" appears 8x,   "xz" appears 7x
3
4BPE picks:    "th" (frequency 500 > 7) ← raw count wins
5WordPiece:    "xz" (score: 7/(10×8) = 0.0875 vs "th": 500/(1000×800) = 0.000625)

WordPiece would merge "xz" first because the tokens almost always co-occur. It's a much stronger signal than the common-but-independent "th".

The ## prefix convention

This convention uses ## to mark subwords that don't start a new word, which preserves word boundaries. For example, tokenizing "unhappiness" produces a list where ## shows which pieces attach to the previous token:

text
1Input: "unhappiness"
2
3WordPiece tokenization:
4  ["un", "##happi", "##ness"]
5  
6The ## tells the model: "this token continues the previous word"
7Without ##: the model couldn't distinguish "un happy" from "unhappy"

Tokenization strategy: greedy longest-match

Unlike BPE, which replays merge rules in order, WordPiece uses a greedy longest-match approach at inference. The function below takes a word and the vocabulary, then repeatedly finds the longest valid subword from the start of the remaining string:

python
1def wordpiece_tokenize(word: str, vocab: set) -> list:
2    """WordPiece greedy longest-match tokenization."""
3    tokens = []
4    start = 0
5    while start < len(word):
6        end = len(word)
7        found = False
8        while start < end:
9            substr = word[start:end]
10            if start > 0:
11                substr = "##" + substr  # Continuation prefix
12            if substr in vocab:
13                tokens.append(substr)
14                found = True
15                break
16            end -= 1
17        if not found:
18            tokens.append("<UNK>")  # Unknown token fallback
19            start += 1
20        else:
21            start = end
22    return tokens

⚠️ Warning: If WordPiece can't tokenize a word at all, it falls back to [UNK] (unknown token). BPE never produces unknown tokens because it falls back to individual bytes.

Head-to-head comparison

---

SentencePiece

SentencePiece, developed by Kudo & Richardson (2018)^[6] at Google, isn't a new tokenization algorithm. It's a framework that makes tokenization truly language-independent by removing the need for pre-tokenization.

💡 Analogy: Think of traditional tokenizers like English-language spell checkers. They work great for English but choke on Japanese or Thai because they assume words are separated by spaces. SentencePiece is more like a universal translator that doesn't assume anything about how languages work. It treats all text as a raw stream of characters, making it equally capable with any writing system.

The language independence problem

Traditional tokenizers like BPE and WordPiece assume text can be split into words first. This works for English, but fails for languages without spaces or with many compound words:

text
1English:  "I love cats" → ["I", "love", "cats"] → subtokenize each
2Japanese: "猫が好きです" → ??? No spaces to split on!
3Thai:     "ฉันรักแมว"    → ??? Also no spaces!
4German:   "Lebensversicherungsgesellschaft" → One giant compound word

SentencePiece solves this by treating the entire input as a raw character stream, including whitespace. It encodes spaces with a special character (▁), creating a continuous sequence that can be tokenized without language-specific rules:

text
1SentencePiece: "I love cats" → "▁I▁love▁cats" → subtokenize directly
2               "猫が好きです"  → "▁猫が好きです"  → subtokenize directly

The ▁ (Unicode U+2581, "lower one eighth block") explicitly represents whitespace as a visible token. This means:

1. •No pre-tokenization needed. It works identically for all languages.
2. •It's perfectly reversible. You can reconstruct the exact original text, including spacing.
3. •Whitespace is semantic. The model can learn that ▁the (word start) differs from the (mid-word continuation).

Two algorithms under one roof

SentencePiece supports two training algorithms:

BPE mode

Standard BPE, but operating on the raw character stream (including ▁). This is what Qwen3.5 and T5 use.

Unigram language model mode

The Unigram model, introduced by Kudo (2018)^[7], takes the opposite approach from BPE:

💡 Analogy: BPE is like building a word from individual letter tiles: you start small and keep gluing pieces together. Unigram is like Michelangelo sculpting David. You start with a massive block of marble (a huge vocabulary) and chisel away the pieces that matter least, revealing the optimal vocabulary hidden inside.

The Unigram algorithm:

1. •Initialize with a large seed vocabulary (e.g., all substrings up to length N, or the top-K most frequent substrings).
2. •Assign probabilities to each token using Expectation-Maximization (EM): $P(\mathbf{x}) = \prod_{i=1}^{M} P(x_i) \quad \text{where each } x_i \text{ is a subword token}$
3. •Find the best segmentation of each training sentence using Viterbi decoding (dynamic programming).
4. •Compute the loss (negative log-likelihood) for each token's removal.
5. •Prune the tokens whose removal least impacts the total corpus likelihood.
6. •Repeat until the target vocabulary size is reached.

💡 Key insight: Unlike BPE (which produces a single deterministic segmentation), Unigram can sample from multiple valid segmentations. This is called subword regularization. During training, you expose the model to different ways of splitting the same word, which acts as a powerful data augmentation technique and improves robustness.^[7]

SentencePiece in practice

Using SentencePiece in production is straightforward with the official Python library. The example below shows how to train a tokenizer on raw text files and then use it to encode and decode new inputs. The training process creates a .model file that can perfectly reverse the tokenization:

python
1import sentencepiece as spm
2
3# Training: language-independent, works on raw text files
4spm.SentencePieceTrainer.train(
5    input='corpus.txt',
6    model_prefix='my_tokenizer',
7    vocab_size=32000,
8    model_type='bpe',           # or 'unigram'
9    character_coverage=0.9995,  # % of characters to cover
10    byte_fallback=True,         # Handle unknown chars via UTF-8 bytes
11)
12
13# Usage
14sp = spm.SentencePieceProcessor(model_file='my_tokenizer.model')
15
16print(sp.encode("Hello world!", out_type=str))
17# ['▁Hello', '▁world', '!']
18
19print(sp.encode("猫が好きです", out_type=str))
20# ['▁猫', 'が', '好き', 'です']
21
22# Perfect reconstruction
23print(sp.decode(['▁Hello', '▁world', '!']))
24# "Hello world!"

Adoption in the industry

SentencePiece's primary advantage is that a single library and model file handles any language without code changes. In BPE mode, it powers models like Qwen3.5, T5, mBART, and ALBERT. In Unigram mode, it's used by XLNet, ALBERT, and some variants of mBART. It remains the standard framework for researchers training custom tokenizers from scratch, especially for multilingual or non-English datasets.

---

Vocabulary size: the critical hyperparameter

Vocabulary size is one of the most consequential design decisions in building a tokenizer. It directly controls the tradeoff between sequence compression (fewer tokens per text) and embedding table size (memory cost).

The tradeoff spectrum

Fertility = average number of tokens per word (or per character for logographic scripts). Lower is better for efficiency.

💡 Key insight: Vocabulary sizes are growing. GPT-2 used 50K tokens, and modern models have expanded to 100K+ tokens. Larger vocabularies reduce sequence length (more fits in context) and improve multilingual support, at the cost of larger embedding tables and slightly slower softmax during inference.

Why bigger isn't always better

It might seem like a bigger vocabulary is always better, but it comes with significant trade-offs:

Advantages of a larger vocabulary

• •✅ Fewer tokens per text (more fits in context window)
• •✅ Better representation of rare words and non-English scripts
• •✅ Lower API cost per character

Disadvantages of a larger vocabulary

• •❌ Larger embedding table (vocab_size × hidden_dim bytes)
• •❌ Rare tokens get less training signal, leading to undertrained embeddings
• •❌ Softmax over a larger vocabulary is slower during inference

The empirical sweet spot for modern LLMs has settled around 32K–128K for primarily English models and 100K–200K for multilingual models.^[8]

---

The tokenization tax: multilingual fairness

One of the most critical aspects of modern LLM performance is tokenization bias, the systematic disadvantage that non-English languages face due to English-centric tokenizer training.

💡 Analogy: Imagine a vending machine that only accepts US quarters. If you're American, a candy bar costs 4 quarters. But if you're Japanese, you first have to exchange your yen into quarters, and the exchange rate is terrible, so the same candy bar costs you 15 quarters. That's the tokenization tax: non-English languages pay more tokens for the same meaning, which means higher API costs, shorter effective context windows, and slower generation.

The problem: unequal fertility

When a tokenizer is trained primarily on English data, non-English text fragments into far more tokens:

Multipliers show token count relative to English for equivalent semantic content.^[9][10]

Real-world consequences

Petrov et al. (2023)^[9] demonstrated that this "tokenization tax" has cascading effects:

🔬 Research insight: A recent study on the "Token Tax" (2025)^[10] found that fertility reliably predicts accuracy on multilingual benchmarks: higher fertility → lower accuracy, consistently across 10 LLMs tested on 16 African languages. Doubling token count quadruples training cost and time.

How industry is addressing it

1. •Larger, balanced vocabularies: OpenAI's o200k_base encoding significantly reduces CJK (Chinese, Japanese, and Korean) and Indic token inflation compared to cl100k_base.
2. •Multilingual training data balancing: Ensuring the tokenizer training corpus has proportional representation across target languages.
3. •Character-coverage thresholds: SentencePiece's character_coverage parameter ensures a minimum percentage of characters get their own token.
4. •Post-hoc vocabulary extension: Adding high-frequency subwords from underrepresented languages after initial training.

---

Unicode normalization: production edge cases

In production systems, raw user input is messy. Engineers must handle Unicode pitfalls to ensure consistent tokenization:

Normalization forms

Python's built-in unicodedata library is the standard tool for handling Unicode variations. The following snippet demonstrates the difference between composed and decomposed forms. It takes a raw, potentially inconsistent string as input and outputs a normalized string (such as via NFKC), ensuring that visually identical characters always produce the same tokens:

python
1import unicodedata
2
3text = "café"  # But which "é"?
4
5# Composed form (NFC): é is a single code point (U+00E9)
6nfc = unicodedata.normalize('NFC', text)   # "café" - 4 chars
7
8# Decomposed form (NFD): é = e + ◌́ (two code points)
9nfd = unicodedata.normalize('NFD', text)   # "café" - 5 chars
10
11# NFKC: Also decomposes compatibility characters
12ligature = "finance"  # fi is a single ligature character
13nfkc = unicodedata.normalize('NFKC', ligature)  # "finance"

🎯 Production tip: If you don't normalize before tokenizing, the same word can produce different token sequences depending on how the Unicode was encoded. Most production tokenizers apply NFKC normalization by default.

Common edge cases in production

---

Modern tokenizer libraries

When implementing tokenization in production, engineers rarely write these algorithms from scratch. The open-source ecosystem provides highly optimized, production-ready libraries that handle the heavy lifting. The choice of library often depends on the target model and the specific deployment environment.

Quick comparison

For OpenAI models, tiktoken is the industry standard due to its raw speed and precise alignment with their APIs. When working with open-source models like Qwen3.5 or Mistral, the Hugging Face tokenizers library provides maximum flexibility. For custom model training, especially in multilingual contexts, SentencePiece remains the go-to framework. The following examples demonstrate how to initialize and use these three major libraries. In each case, the library takes raw text as input and outputs a sequence of integer token IDs that a model can process, as well as providing methods to decode those IDs back into readable text:

python
1# ─── tiktoken (OpenAI) ──────────────────────────
2# Rust-based, optimized for OpenAI models
3import tiktoken
4
5enc = tiktoken.get_encoding("o200k_base")    # 200K vocab, used by newer OpenAI tokenizers
6tokens = enc.encode("Hello, world!")          # [13225, 11, 2375, 0]
7text = enc.decode(tokens)                     # "Hello, world!"
8num_tokens = len(tokens)                      # 4
9
10# ─── HuggingFace tokenizers ────────────────────
11# Rust-based, supports all architectures
12from transformers import AutoTokenizer
13
14tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B")
15out = tok("Hello, world!", return_tensors="pt")
16# out.input_ids: tensor([[8774, 11, 1917, 0]])
17
18# ─── SentencePiece ─────────────────────────────
19# C++-based, for training custom tokenizers
20import sentencepiece as spm
21
22sp = spm.SentencePieceProcessor(model_file='my_tokenizer.model')
23pieces = sp.encode("Hello, world!", out_type=str)
24# ['▁Hello', ',', '▁world', '!']

---

The complete algorithm comparison

To summarize the differences, it helps to look at these three approaches side-by-side. The key distinctions are their directionality (bottom-up merging vs. top-down pruning) and their decision criteria (raw frequency vs. statistical likelihood).

---

Specialized case: tokenization for code

Code tokenization requires specific handling distinct from natural language because code has strict syntax rules where whitespace and punctuation matter deeply.

Why standard NLP tokenizers fail on code

Standard BPE or WordPiece trained on English text often fails on code:

1. •Whitespace sensitivity: In Python, indentation is semantic. A tokenizer that treats multiple spaces as one token or ignores them breaks the code's logic.
2. •Long identifiers: getUserByEmailAddress might be split into ['get', 'User', 'By', 'Email', 'Address'] (good) or ['getU', 'serB', 'yEm', 'ailA', 'ddress'] (bad). CamelCase splitting is crucial.
3. •Operators: Symbols like ===, !==, ->, :: should be atomic tokens, not fragmented into ['=', '=', '='].
4. •Numerical literals: Floating point numbers (3.14159) shouldn't be split arbitrarily.

Modern solutions (OpenAI o200k_base & StarCoder)

Recent tokenizers like OpenAI's o200k_base explicitly add code-specific tokens to the vocabulary to reduce fragmentation. The example below compares how a smaller tokenizer and the larger, code-optimized one handle the same Python snippet. The larger tokenizer preserves the function name as a single unit, avoiding the fragmentation that would otherwise occur:

python
1# Example: Tokenizing code with a smaller tokenizer vs OpenAI's o200k_base
2
3code = "    def calculate_pi():"
4
5# Smaller tokenizer (~50K vocab): fragments the function name
6# ['    ', 'def', ' calculate', '_', 'pi', '():'] (6 tokens)
7
8# o200k_base (~200K vocab): preserves common identifiers
9# ['    ', 'def', ' calculate_pi', '():'] (4 tokens)

💡 Key insight: Reducing the number of tokens per line of code directly increases the context window's effective size for repository-level reasoning. This is why models like StarCoder and Code Llama use specialized BPE training on GitHub data.

---

Key takeaways

• •The Core Problem: Tokenization bridges raw text and model computation. The fundamental challenge is choosing a granularity between characters and words that balances sequence length and vocabulary size.
• •Algorithm Comparison:
  • •BPE: Frequency-based merging, bottom-up (GPT series, Qwen3.5).
  • •WordPiece: Likelihood-based merging (PMI), bottom-up (BERT).
  • •Unigram: Likelihood-based pruning, top-down (T5, ALBERT).
• •SentencePiece's Innovation: Achieves language independence through raw-text processing and the ▁ whitespace marker, supporting both BPE and Unigram modes without pre-tokenization.
• •Vocabulary Tradeoffs: Larger vocabularies (50K → 200K) reduce sequence length and improve multilingual support but increase embedding memory costs.
• •Production Reality: Real-world tokenization requires handling Unicode normalization (NFC/NFKC) and mitigating the "tokenization tax" that increases costs and latency for non-English languages.

Common misconceptions

• •Confusing BPE with WordPiece: BPE merges based on raw frequency; WordPiece merges based on likelihood improvement (PMI).
• •Misunderstanding ##: This is a specific convention of WordPiece (and BERT), not a universal property of subword tokenization.
• •SentencePiece as an Algorithm: SentencePiece is a library/framework that implements BPE and Unigram; it isn't an algorithm itself.
• •Byte-Level BPE & UNK: Byte-level BPE (used in GPT-2+) generally can't produce unknown tokens because it falls back to raw bytes.
• •Ignoring Multilingual Cost: Vocabulary choice isn't just about English performance; it determines the cost/latency tier for all other supported languages.