BPE, WordPiece, and SentencePiece

A customer types, "My refund for order 48291 still isn't here." Your support model can't read that sentence directly. It receives integer IDs, and a tokenizer decides which pieces of the message get those IDs.

In the previous lesson, you saw why learning systems need representations that can improve with data and compute. Tokenization is the first such representation for text: a fixed contract that turns new wording, languages, and symbols into model input. You'll compare Byte Pair Encoding (BPE), WordPiece, and SentencePiece as you build that contract, break it, and learn what to measure before shipping it.

An embedding is a vector associated with a token ID. The next lesson teaches those vectors. For now, keep one rule in mind: ID 421 has no universal meaning. It only means something when paired with the exact tokenizer artifact that produced it.

Choose pieces between characters and words

A word-level tokenizer could keep refund as one item, but it needs a policy for every misspelling, product name, and new tracking code. A character-level tokenizer never runs out of pieces, but it turns short messages into long sequences. Subword tokenization keeps frequent fragments whole while retaining small fallback pieces for rare text.

Consider this ticket:

The first lab makes that tradeoff concrete. The subword split is written by hand because you haven't trained a tokenizer yet. It gives you a budget to compare against character and whitespace-word baselines.

The word count looks smallest, but it hides the hard question: what happens when order48291 was never in the vocabulary? Subwords answer that question by learning common fragments and retaining smaller pieces for the rest.

Train byte pair encoding from counts

Byte Pair Encoding (BPE) builds a vocabulary from repeated adjacent pieces. Sennrich, Haddow, and Birch applied BPE subword units to open-vocabulary neural machine translation: start from small symbols, merge frequent adjacent pairs, and reuse the resulting pieces for rare words.^[1]

For teaching, start with characters inside pre-separated ticket terms. A production tokenizer has extra decisions about whitespace, bytes, and normalization, but the merge loop is the important first mechanism.

Suppose a delivery-support corpus contains these term counts:

At the start, s + h appears seven times: three in ship, two in shipping, and two in shop. Merge it into sh. On the updated corpus, sh + i appears five times and can become shi; then shi + p becomes ship.

This miniature trainer stores words as tuples of current pieces, counts adjacent pairs, merges the winner everywhere, and prints the first three learned rules.

The result isn't a linguistic analysis. BPE doesn't know that ship relates to logistics. It knows only that a boundary occurs often enough to compress.

Replay merges on a new term

Training chooses an ordered merge list once. Encoding a new input doesn't recount a new corpus; it replays those learned rules. That distinction matters in production because the tokenizer must stay fixed with the model.

The next lab takes the three rules learned above and applies them to new terms. shipper benefits from the common stem, while shopper gets only the sh merge because the corpus never earned shop as one piece in the first three steps.

Fall back to bytes when characters are new

Character-starting BPE still needs an unknown-token policy for characters absent from its base vocabulary. GPT-2 used a byte-level BPE variant: it begins from representations for bytes, then learns merges over that base, which lets any UTF-8 input remain representable without an unknown character token.^[2]

This lab doesn't train merges; it demonstrates the safety net beneath byte-level BPE. A return message containing Japanese characters and an emoji is still reversible through raw UTF-8 byte values.

Byte coverage prevents an unrepresentable character. It doesn't promise compact tokenization: if the training data rarely covers a script or emoji sequence, several bytes may remain separate pieces.

WordPiece chooses vocabulary differently

WordPiece appeared in Google's Japanese and Korean voice-search work and later became familiar through BERT's tokenizer.^[3][4] Like BPE, it creates reusable pieces. Unlike simple frequency-based BPE, the original WordPiece description selects vocabulary additions to improve a language-model likelihood objective.

Exact training recipes aren't fully specified by the short original paper, and library trainers can differ. A useful classroom proxy is an association score:

$$\operatorname{score}(a,b) = \frac{\operatorname{count}(ab)} {\operatorname{count}(a)\operatorname{count}(b)}$$

Here, count(ab) is how often two neighboring pieces occur together; the denominator penalizes pieces that occur frequently in many other contexts. This formula teaches why a less frequent but tightly associated pair could be attractive. Treat it as intuition for WordPiece's likelihood motivation, not as the original implementation specification.

At encoding time, BERT-style WordPiece uses continuation pieces such as ##ing and a greedy longest-match lookup. A piece beginning with ## continues the current word rather than beginning a new word.

The next lab implements that lookup for support vocabulary. It always tries the longest valid piece from the current cursor position.

Expose the unknown-token failure

Standard WordPiece can't necessarily spell a word from arbitrary bytes. If no valid segmentation reaches the end of a word, BERT-style tokenization emits [UNK] for that word. That loses distinctions between two different unseen strings.

This failure test uses the same algorithm with a missing ##bot continuation. The fix isn't to silently map new strings to a known ID. You must use the model's tokenizer contract, or explicitly change the vocabulary and corresponding model parameters as a training decision.

SentencePiece treats boundaries as part of the artifact

Many subword pipelines first split a sentence into word-like units, then segment inside each unit. That assumption is awkward for text where spaces don't mark each word. SentencePiece is a tokenizer and detokenizer framework that trains directly from raw sentences instead of requiring pre-tokenized word sequences.^[5]

SentencePiece commonly makes spaces visible as ▁, so Orders ship late becomes a stream like ▁Orders▁ship▁late before final pieces are chosen. Decoding targets the normalized input string, not necessarily the original raw byte sequence, because normalization can fold equivalent or compatibility forms.

SentencePiece supports BPE, and it also supports the Unigram language model algorithm proposed with subword regularization. Unigram starts with many candidate pieces, assigns probabilities, removes pieces that contribute least to corpus likelihood, and can sample multiple valid segmentations during model training.^[6]

That sampling option matters because a word such as refundable can have several legal piece boundaries. Training on more than one segmentation can make the downstream model less dependent on a single boundary choice. For deterministic serving, the tokenizer can still select its highest-probability segmentation.

The official library makes the artifact visible. This lab trains a small Unigram SentencePiece model on raw support messages, encodes an unseen request, and proves that its decoded form matches the tokenizer's normalized text.

Vocabulary size spends parameters to save positions

Every added vocabulary entry can compress a recurring string into fewer tokens. It also adds an embedding row. If the output projection isn't tied to the input embedding matrix, it adds another row there too.

For a vocabulary of size $V$ and hidden dimension $d$, an input embedding matrix contains $Vd$ parameters. With float16 weights, each parameter takes two bytes. A second untied output matrix doubles that vocabulary-dependent memory.

Use numbers before making a design argument. This lab compares hypothetical tokenizer vocabularies for a model with hidden dimension 4096; it calculates embedding memory rather than assuming a larger vocabulary is free.

Sequence compression must be measured too. A vocabulary can shorten common English refund requests and still fragment another script or your TypeScript repository badly. This is why tokenizer design is an evaluation problem, not a race to the largest V.

Audit language and code token budgets

Fertility is a token-length measure: how many tokenizer pieces a unit of text needs. For a product with international customers, compare parallel messages across target languages instead of using English traffic alone. Petrov et al. measured translated text and found tokenizer-length disparities as large as 15 times for some language and tokenizer combinations, with implications for cost, latency, and available context.^[7]

Code belongs in the same audit. Identifiers, indentation, and operators are all model input. A tokenizer trained with limited code coverage may split familiar repository patterns into many pieces, leaving less room for files, tests, and error logs.

The next lab uses one named tokenizer artifact to measure several fixtures. The sample isn't a language-quality study; its job is to give you a repeatable audit shape. Replace the fixture messages with reviewed parallel translations and repository files for a real decision.

Don't read a single sample as a ranking of languages. Build a locale-aware test set, record tokenizer version, compare distribution summaries, and then check downstream task quality.

Make Unicode policy explicit

Two strings can look identical while holding different Unicode code points. For example, café may contain one composed é or the sequence e plus a combining accent. Without a stable policy, cache keys, token counts, and filter behavior can disagree across clients.

Normalization Form C (NFC) composes canonically equivalent forms without folding broad compatibility distinctions. Normalization Form Compatibility Composition (NFKC) also folds compatibility characters, such as the fi ligature into fi. Python exposes both through unicodedata.normalize; choosing between them is a product policy decision, not an automatic cleanup rule.

This final lab proves canonical equivalence and highlights the compatibility choice. For user-visible support messages, you might choose NFC first and add separate security checks for invisible or confusable characters. Another product may deliberately choose NFKC after deciding the information loss is acceptable.

Tokenizer behavior must be versioned with this policy. If one service normalizes with NFC and another silently folds with NFKC, they can send different IDs to the same model or generate different cache keys for a ticket that looks unchanged.

Compare algorithms without mixing contracts

You can now read the common tokenizer families as design choices rather than names to memorize.

Production review checklist

What you can now build

You started with a refund sentence and finished with a tokenizer review harness. You can now:

1. Explain why subwords balance sequence length against vocabulary coverage.
2. Train and replay a minimal BPE merge table.
3. Explain why byte fallback preserves representability but doesn't guarantee compactness.
4. Implement WordPiece longest-match encoding and reproduce its unknown-token failure.
5. Train a raw-text SentencePiece Unigram tokenizer and distinguish the framework from its segmentation algorithms.
6. Calculate vocabulary-dependent embedding memory.
7. Audit token length across locales and code fixtures without overstating one sample.
8. Choose and test a Unicode normalization policy.

Mastery check

Evaluation rubric

• Foundational: Given a five-word corpus, you can calculate one BPE winner by hand and apply it to updated pieces.
• Intermediate: You can implement BPE replay and WordPiece longest-match segmentation, then explain why their failure behavior differs.
• Intermediate: You can show the difference between SentencePiece BPE and SentencePiece Unigram without calling SentencePiece a merge algorithm.
• Advanced: You can present a tokenizer audit with locale fixtures, code fixtures, normalization policy, vocabulary memory cost, and artifact versioning.

Follow-up questions

Common pitfalls

• Retraining during inference: Pair frequencies are counted while building BPE rules, not while serving each customer message. Serving must replay fixed rules.
• Describing the WordPiece proxy as its spec: The association score clarifies the intuition. The original method is likelihood-driven, and implementations can vary.
• Calling SentencePiece a fourth algorithm: SentencePiece packages raw-text handling and can host BPE or Unigram models.
• Assuming byte fallback means fair multilingual cost: Coverage avoids unknown characters; it doesn't make segment lengths equal.
• Normalizing without a policy: NFC and NFKC solve different problems. Compatibility folding can discard distinctions your product intended to preserve.