BPE, WordPiece, and SentencePiece

A developer writes const tokenCount = encodePrompt(prompt).length;. A model can't read that code or any other text directly. It receives integer IDs, and a tokenizer decides which pieces of the string get those IDs.

In the previous lesson, you saw why learning systems need representations that can improve with data and compute. Tokenization is the first such representation for text: a fixed contract that turns new wording, languages, and symbols into model input. You'll compare Byte Pair Encoding (BPE), WordPiece, and SentencePiece as you build that contract, break it, and learn what to measure before serving it.

An embedding is a vector associated with a token ID. The next lesson teaches those vectors. For now, keep one rule in mind: ID 421 has no universal meaning. It only means something when paired with the exact tokenizer artifact that produced it.

Choose pieces between characters and words

A word-level tokenizer could keep tokenizer as one item, but it needs a policy for every rare identifier, version string, and prompt marker. A character-level tokenizer never runs out of pieces, but it turns short strings into long sequences. Subword tokenization keeps frequent fragments whole while retaining small fallback pieces for rare text.

Consider this prompt fixture:

The first lab makes that tradeoff concrete. The subword split is written by hand because you haven't trained a tokenizer yet. Use its token count as a budget to compare against character and whitespace-word baselines.

The word count looks smallest, but it hides the hard question: what happens when prompt128k was never in the vocabulary? Subwords answer that question by learning common fragments and retaining smaller pieces for the rest.

Train byte pair encoding from counts

Byte Pair Encoding (BPE) builds a vocabulary from repeated adjacent pieces. Sennrich, Haddow, and Birch applied BPE subword units to open-vocabulary neural machine translation: start from small symbols, merge frequent adjacent pairs, and reuse the resulting pieces for rare words.^{[1]Reference 1Neural Machine Translation of Rare Words with Subword Units.https://arxiv.org/abs/1508.07909}

For teaching, start with characters inside pre-separated code and text terms. A production tokenizer has extra decisions about whitespace, bytes, and normalization, but the merge loop is the important first mechanism.

Suppose a code-text corpus contains these term counts:

At the start, c + o appears seven times: three in code, two in coder, and two in cope. Merge it into co. On the updated corpus, co + d and d + e are tied at five. The toy trainer encounters co + d first, so it becomes cod; a production trainer needs an explicit deterministic tie-break. Recounting again makes cod + e the next winner at five, producing code.

This miniature trainer stores words as tuples of current pieces, counts adjacent pairs, merges the winner everywhere, and prints the first three learned rules.

The result isn't a linguistic analysis. BPE doesn't know that code relates to programming. It knows only that a boundary occurs often enough to compress.

Replay merges on a new term

Training chooses an ordered merge list once. Encoding a new input doesn't recount a new corpus; it replays those learned rules. That distinction matters in production because the tokenizer must stay fixed with the model.

The next lab takes the three rules learned above and applies them to new terms. coder benefits from the common stem, while coper gets only the co merge because the corpus never earned cope as one piece in the first three steps.

During BPE encoding, why don't you count pairs in the user's new message?

Answer

Pair counting belongs to tokenizer training. At inference time the model needs the same stable input contract used during its training, so the encoder replays the fixed ordered merge list.

Use bytes as a complete base alphabet

Character-starting BPE still needs an unknown-token policy for characters absent from its base vocabulary. GPT-2 used a byte-level BPE variant: its base alphabet represents bytes, then learned merges build larger pieces over that base. Because any Unicode string has a UTF-8 byte representation, every input remains representable without an unknown character token.^{[2]Reference 2Language Models are Unsupervised Multitask Learners.https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf}

Some tokenizers use an explicit byte fallback only when ordinary pieces can't encode an input. That has the same coverage goal but isn't the same mechanism as GPT-2's byte-level starting alphabet. This lab doesn't train merges. It isolates the common foundation: a code comment containing Japanese characters and an emoji is reversible through raw UTF-8 byte values.

Byte coverage prevents an unrepresentable character. It doesn't promise compact tokenization: if the training data rarely covers a script or emoji sequence, several bytes may remain separate pieces.

WordPiece chooses vocabulary differently

WordPiece appeared in Google's Japanese and Korean voice-search work and later became familiar through BERT's tokenizer.^{[3]Reference 3Japanese and Korean Voice Search.https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37842.pdf[4]Reference 4BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} Like BPE, it creates reusable pieces. Unlike simple frequency-based BPE, the original WordPiece description selects vocabulary additions to improve a language-model likelihood objective.

Exact training recipes aren't fully specified by the short original paper, and library trainers can differ. A useful classroom proxy is an association score:

$$\operatorname{score}(a,b) = \frac{\operatorname{count}(ab)} {\operatorname{count}(a)\operatorname{count}(b)}$$

Here, count(ab) is how often two neighboring pieces occur together; the denominator penalizes pieces that occur frequently in many other contexts. This formula teaches why a less frequent but tightly associated pair could be attractive. Treat it as intuition for WordPiece's likelihood motivation, not as the original implementation specification.

At encoding time, BERT-style WordPiece uses continuation pieces such as ##ing and a greedy longest-match lookup. A piece beginning with ## continues the current word rather than beginning a new word.

The next lab implements that lookup for tokenizer vocabulary. It always tries the longest valid piece from the current cursor position.

Expose the unknown-token failure

Standard WordPiece can't necessarily spell a word from arbitrary bytes. If no valid segmentation reaches the end of a word, BERT-style tokenization emits [UNK] for that word. That loses distinctions between two different unseen strings.

This failure test uses the same algorithm with a missing ##bot continuation. The fix isn't to silently map new strings to a known ID. You must use the model's tokenizer contract, or explicitly change the vocabulary and corresponding model parameters as a training decision.

SentencePiece treats boundaries as part of the artifact

Many subword pipelines first split a sentence into word-like units, then segment inside each unit. That assumption is awkward for text where spaces don't mark each word. SentencePiece is a tokenizer and detokenizer framework that trains directly from raw sentences instead of requiring pre-tokenized word sequences.^{[5]Reference 5SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing.https://arxiv.org/abs/1808.06226}

SentencePiece commonly makes spaces visible as ▁, so Token cache warm becomes a stream like ▁Token▁cache▁warm before final pieces are chosen. Decoding targets the normalized input string, not necessarily the original raw byte sequence, because normalization can fold equivalent or compatibility forms.

SentencePiece supports BPE, and it also supports the Unigram language model algorithm proposed with subword regularization. Unigram starts with many candidate pieces, assigns probabilities, removes pieces that contribute least to corpus likelihood, and can sample multiple valid segmentations during model training.^{[6]Reference 6Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates.https://arxiv.org/abs/1804.10959}

That sampling option matters because a word such as tokenized can have several legal piece boundaries. Training on more than one segmentation can make the downstream model less dependent on a single boundary choice. For deterministic serving, the tokenizer can still select its highest-probability segmentation.

The official library makes the artifact visible. This lab trains a small Unigram SentencePiece model on raw tokenizer-related text, encodes an unseen request, and proves that its decoded form matches the tokenizer's normalized text.

The default SentencePiece normalization folds the compatibility ligature fi into fi. The visible ▁ marker participates in segmentation, then decoding restores spaces. That's why the decoded output matches normalized text rather than the original raw code-point sequence.

Is SentencePiece a fourth merge algorithm alongside BPE and WordPiece?

Answer

No. SentencePiece is a raw-text tokenizer and detokenizer framework. It can train a BPE model or a Unigram model, while also packaging its normalization and boundary-handling policy.

Vocabulary size spends parameters to save positions

Every added vocabulary entry can compress a recurring string into fewer tokens. It also adds an embedding row. If the output projection isn't tied to the input embedding matrix, it adds another row there too.

For a vocabulary of size $V$ and hidden dimension $d$, an input embedding matrix contains $Vd$ parameters. With float16 weights, each parameter takes two bytes. A second untied output matrix doubles that vocabulary-dependent memory.

Use numbers before making a design argument. This lab compares hypothetical tokenizer vocabularies for a model with hidden dimension 4096; it calculates embedding memory rather than assuming a larger vocabulary is free.

Sequence compression must be measured too. A vocabulary can shorten common English tokenizer prompts and still fragment another script or your TypeScript repository badly. Tokenizer design is an evaluation problem, not a race to the largest V.

Audit language and code token budgets

Fertility is a token-length measure. At word level, fertility is the average number of tokenizer pieces needed per word. For parallel-message audits, you can also compare total token count or a locale-to-baseline ratio for equivalent text. For a product with international users, compare parallel messages across target languages instead of using English traffic alone. Petrov et al. measured translated text and found tokenizer-length disparities as large as 15 times for some language and tokenizer combinations, with implications for cost, latency, and available context.^{[7]Reference 7Language Model Tokenizers Introduce Unfairness Between Languages.https://arxiv.org/abs/2305.15425}

Code belongs in the same audit. Identifiers, indentation, and operators are all model input. A tokenizer trained with limited code coverage may split familiar repository patterns into many pieces, leaving less room for files, tests, and error logs. Human-readable splits are useful debugging clues, not proof of model quality. Measure token totals and validate downstream code tasks.

The next lab uses one named tokenizer artifact to measure several fixtures. The sample isn't a language-quality study; its job is to give you a repeatable audit shape. Replace the fixture messages with reviewed parallel translations and repository files for a real decision.

Don't read a single sample as a ranking of languages. Build a locale-aware test set, record tokenizer version, compare distribution summaries, and then check downstream task quality.

Make Unicode policy explicit

Two strings can look identical while holding different Unicode code points. For example, café may contain one composed é or the sequence e plus a combining accent. Without a stable policy, cache keys, token counts, and filter behavior can disagree across clients.

Normalization Form C (NFC) composes canonically equivalent forms without folding broad compatibility distinctions. Normalization Form Compatibility Composition (NFKC) also folds compatibility characters, such as the fi ligature into fi. Python exposes both through unicodedata.normalize; choosing between them is a product policy decision, not an automatic cleanup rule.

This final lab proves canonical equivalence and makes the compatibility choice visible. For user-visible prompts, you might choose NFC first and add separate security checks for invisible or confusable characters. Another product may deliberately choose NFKC after deciding the information loss is acceptable.

Tokenizer behavior must be versioned with this policy. If one service normalizes with NFC and another silently folds with NFKC, they can send different IDs to the same model or generate different cache keys for text that looks unchanged.

Compare algorithms without mixing contracts

The common tokenizer families are design choices, not names to memorize.

Production review checklist

Mastery check

Evaluation rubric

• Foundational: Given a five-word corpus, you can calculate one BPE winner by hand and apply it to updated pieces.
• Intermediate: You can implement BPE replay and WordPiece longest-match segmentation, then explain why their failure behavior differs.
• Intermediate: You can show the difference between SentencePiece BPE and SentencePiece Unigram without calling SentencePiece a merge algorithm.
• Advanced: You can present a tokenizer audit with locale fixtures, code fixtures, normalization policy, vocabulary memory cost, and artifact versioning.

Follow-up questions

Why must a deployed model keep the same tokenizer artifact used during training?

Answer

The model learned embeddings and output probabilities for specific token IDs. A different vocabulary, merge list, normalization policy, or ID mapping changes the integers representing the text, so the learned parameters no longer receive the intended inputs.

What does byte fallback solve, and what does it leave unsolved?

Answer

Byte fallback makes every UTF-8 input representable without an unknown character token. It doesn't guarantee good compression or equal token cost across languages; those properties require corpus coverage and measurement.

Why isn't a low token count on one English prompt enough to choose a tokenizer?

Answer

A production workload includes many message forms, locales, and possibly code or documents. Measure token distributions and downstream quality on representative fixtures, then account for vocabulary memory and normalization behavior.

Common pitfalls

• Retraining during inference: Pair frequencies are counted while building BPE rules, not while serving each prompt. Serving must replay fixed rules.
• Conflating byte-level BPE with byte fallback: GPT-2 starts from byte representations. Other tokenizers may invoke explicit byte fallback only when ordinary pieces can't encode an input. Both preserve coverage, but they aren't the same mechanism.
• Describing the WordPiece proxy as its spec: The association score clarifies the intuition. The original method is likelihood-driven, and implementations can vary.
• Calling SentencePiece a fourth algorithm: SentencePiece packages raw-text handling and can host BPE or Unigram models.
• Assuming byte fallback means fair multilingual cost: Coverage avoids unknown characters; it doesn't make segment lengths equal.
• Normalizing without a policy: NFC and NFKC solve different problems. Compatibility folding can discard distinctions your product intended to preserve.