Subword Tokenisation: BPE, WordPiece and Unigram Models

Why whole-word vocabularies run out of room

A fixed word vocabulary assigns a separate entry to every stored spelling. Inflected forms, compounds, product codes and typos create new spellings continually. An unknown-word token collapses all excluded words into one symbol, discarding distinctions that the model might need.

Characters have broad coverage but produce long sequences. Subwords occupy the space between these choices: frequent strings become single units while unfamiliar strings can be assembled from smaller pieces. Coverage still depends on the base alphabet and fallback mechanism. A character-based vocabulary that never included a script does not magically encode that script.

A BPE training step, by hand

Use a toy word inventory with frequencies: low occurs five times, lower twice, and new three times. Start from characters and a word-boundary symbol. Count adjacent symbol pairs weighted by word frequency.

The most frequent pair is w + boundary, so merge it into one symbol. The representations of low and new change, while lower keeps its internal w. Recompute affected pairs and continue until the merge budget is reached. Ties require a deterministic policy.

At inference, apply the learned merge ranking to the new string. Do not re-estimate frequencies from the evaluation text. Training chooses the code; inference uses it. The boundary convention also matters: an end-of-word marker can distinguish low as a complete word from the same substring inside another word.

Three families, three decisions

BPE specifies a greedy merge construction. WordPiece is commonly associated with building a piece vocabulary and greedily choosing the longest matching piece at each position; training implementations differ, so the model name alone is not a complete algorithm specification. Unigram language-model tokenisation assigns probabilities to pieces and scores a segmentation by the product of its piece probabilities.

For unigram pieces \(z_1,\ldots,z_m\) that concatenate to a word \(x\),

A dynamic program can find the best segmentation. The method can also sample segmentations during training, exposing a downstream model to several decompositions of the same string. That regularises the representation without requiring a change in the underlying text.

Vocabulary size has two costs

A larger vocabulary generally reduces token count for familiar text but increases the embedding table and output classifier. With vocabulary size \(V\) and dimension \(d\), an embedding matrix contains \(Vd\) parameters. A smaller vocabulary saves those parameters but may expand the sequence, increasing the number of prediction steps and the work of sequence processing.

Measure the tradeoff on the actual language mix. If one language needs twice as many tokens per character, a fixed token window gives it less document context. Report fragmentation by language or domain instead of only an aggregate mean.

Reversibility and offsets

A reversible tokeniser can reconstruct the representation it was given. It cannot reconstruct distinctions removed earlier by lowercasing or Unicode normalisation. Keep the raw text separately if exact round trips matter.

For span tasks, carry token-to-character offsets and mask any training positions that do not correspond to annotated units. For language-model evaluation, compare perplexities only under compatible tokenisation; changing the units changes the denominator.

References

1. Sennrich, Haddow and Birch. Neural Machine Translation of Rare Words with Subword Units. ACL, 2016.
2. Kudo, T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. ACL, 2018.