Second Foundations post in ML & AI. How the Transformer Actually Works started from token embeddings as if they were given — this post is about the step before that: turning raw text into the discrete units a model actually operates on.

Why not just use words, or just characters

Both extremes have real problems:

  • Word-level vocabularies get enormous — every inflection, misspelling, and compound needs its own entry — and any word the vocabulary didn't happen to include at training time becomes unrepresentable.
  • Character-level vocabularies are tiny, but sequences get long: a sentence that's a handful of words becomes dozens of characters, and since attention's cost grows with sequence length, that's a real cost, not just an inconvenience.

Subword tokenization is the middle ground: common words stay as a single token, rare or unseen words get broken into smaller, still meaningful pieces instead of falling back to an unknown-word token.

Byte-Pair Encoding

BPE was originally a data-compression algorithm, adapted for this purpose by Sennrich, Haddow, and Birch (2015). The algorithm is genuinely simple:

  1. Start with every word split into individual characters.
  2. Count every adjacent pair of symbols across the whole corpus.
  3. Merge the single most frequent pair into a new symbol.
  4. Repeat, now treating that merged pair as one symbol, until reaching a target vocabulary size.

A small worked example — merging just the word "lowest", where its frequent characters in a toy corpus of { low, lowest, newer } drive which pairs merge first:

Diagram showing "lowest" split into six characters, then merged step by step: first "w"+"e" into "we", then "l"+"o" into "lo", ending at four tokens

w+e merges first because, counted across the whole corpus, that pair is more frequent than any other (it shows up in both "lowest" and "newer"). Then l+o merges, because among the remaining pairs it's now the most common. Keep going, and frequent whole words eventually become single tokens, while rare ones stay split into pieces that occurred often enough elsewhere to have earned their own symbol.

Byte-level BPE: no unknown-token fallback, ever

Running BPE over Unicode characters still leaves a gap: a character the training corpus never saw (an unusual symbol, an emoji, text in a script the corpus barely covered) still has nowhere to go. GPT-2 (Radford et al., 2019) closes that gap by running BPE over raw UTF-8 bytes instead of characters — a fixed base vocabulary of exactly 256 symbols, one per possible byte value. Every possible input, in any language, is expressible as some sequence of bytes, so there's no input this can fail to tokenize. GPT-2's tokenizer starts from that 256-symbol base and merges up to a final vocabulary of 50,257 tokens — no <unk> token needed anywhere in the vocabulary.

(BERT uses a related but distinct scheme, WordPiece, which merges pairs by a likelihood criterion instead of raw frequency — same overall idea, different rule for choosing what to merge next.)

Why this is worth knowing, not just a preprocessing detail

Tokenization quietly decides things people usually attribute to the model itself:

  • Cost and context window are measured in tokens, not characters or words — and since BPE vocabularies are typically trained on corpora dominated by one language, the same sentence can cost noticeably more tokens in a language underrepresented in that training data, purely because fewer of its subwords earned a spot as a single merged token.
  • Character-level tasks are hard for LLMs for a structural reason, not a reasoning failure: if "strawberry" is a single token (or two or three subword tokens), the model never directly sees the individual letters inside it — asking it to count or manipulate characters is asking it to reason about something it was never given as an input in the first place.

Tokenization is the layer that decides what a model's "unit of thought" even is — everything downstream, including the attention mechanism from the previous post, operates on whatever these merges produced.

References

Merge-step diagram above is original artwork made for this post.