BPE Merges, Playable

EP 01

The merge machine — watch a vocabulary being born

Here is a toy corpus of 20 words, exploded into single characters (the • marks end-of-word). The table shows every neighboring pair and how often it occurs. Press merge: the top pair fuses into a new symbol, everywhere at once, and the vocabulary grows by one. This is the entire BPE training algorithm — there is nothing else.

Same-colored tiles are the same symbol. Watch for t+h → th, then th+e → the — but the very first merges may surprise you: the algorithm only cares about raw pair counts.

Why it matters: this loop, run ~100,000 times on terabytes of text instead of 22 words, produced the actual vocabulary inside GPT. Good case: "the" fuses into one symbol within a handful of merges because it's everywhere. Bad case: keep pressing — once every remaining pair occurs only once, the machine stalls, and the rare words are left in pieces forever.

EP 02

The merge dial — from alphabet soup to whole words

Now a bigger corpus (~130 words), trained the same way at page load. Drag the dial to choose how many merges the tokenizer is allowed to use, and watch one sentence re-tokenize live. The curve underneath is the sentence's token count at every dial position — computed from the real merges, not drawn by hand.

merges 0

The shape of the curve is the whole economics of tokenization: the first dozen merges slash the token count (good case — cheap, huge wins), then the curve goes flat (bad case — every extra vocabulary slot buys almost nothing). Real labs pick vocabulary size by staring at exactly this diminishing-returns curve, at 100k scale.

EP 03

Token birthdays — frequent words get names first

A word is "born" as a token the moment enough merges chain together to fuse it into a single symbol. Because merges are ranked by frequency, birth order is a frequency ranking: the corpus's favorite words get names first. The leaderboard below is computed live from the trained merges. Then type any word and ask when — or whether — it was ever born.

check a word

"Never born" doesn't mean unusable — the word still tokenizes fine, just as several pieces. In GPT's real vocabulary, " the" was among the earliest merges; your surname probably never got born at all.

Why it matters: token IDs are frequency fossils. Common words are short, early, single tokens; rare ones are late or never. That's why LLMs are subtly better at everything common — the popular stuff got both a dedicated symbol and the most training data behind it.

EP 04

The shredder — feed it a word it has never met

The merges are frozen now — training is over, exactly like a shipped model. Feed the tokenizer words it knows, words it half-knows, and words that don't exist. It never refuses (any spelling can be covered by characters), but watch a made-up word get destroyed into confetti while "garden" sails through as one piece.

Notice "gardens": never in the corpus, yet it reuses the fused "garden" symbol plus scraps — BPE generalizes to near words for free. Pure inventions get nothing but character-level shrapnel.

Why it matters: this is what happens inside GPT when it meets new slang, your product name, or a typo — the word arrives as many low-information fragments the model rarely saw together. It's a real reason models stumble on novel names, and why brands with "tokenizer-friendly" names get slightly better treatment for free.

EP 05

Goldilocks vocabulary — characters, words, or something in between

Why merge at all? Compare the three ways to slice the same sentence. Characters: tiny vocabulary, absurdly long sequences. Whole words: short sequences — until an unseen word appears and becomes a dead ❓UNK, its meaning erased. BPE: the compromise that never says UNK. Edit the sentence — try adding made-up words — and watch all three respond.

The design space in one sentence: character models waste the network's attention on spelling, word models amputate anything new, and BPE sits in the Goldilocks zone — which is why essentially every modern LLM (GPT, Claude, Llama) uses BPE or a close cousin. The vocabulary size (~100k) is just where someone stopped turning the dial from EP 02.