Why language models read tokens, not words

Tokenization & decoding · Part 1 of 8

Hand a model the word “unbelievable” and ask it to read. What should it look at: the letters, the whole word, or something in between? That choice, made before any learning starts, shapes everything that follows.

Option 1: letters

Use single characters and the vocabulary is tiny, a few hundred symbols at most, and no word is ever unknown. The price is length. A page of text becomes thousands of positions, and a model has to rebuild every word from its letters before it can think about meaning. Longer sequences also mean more work for attention.

Option 2: whole words

Use one slot per word and sequences become short. But now every form needs its own slot: walk, walks, walked, walking. Names, typos and new words are missing from the list, so they become a useless “unknown” token. Rare words get few training examples and are poorly understood.

Option 3: pieces of words

Subword tokenization splits the difference. Common words stay as one token; rare words are cut into pieces that were useful elsewhere. Nothing is ever unknown, because any word can at worst be spelled out. The pieces are not always neat, as you can see below, but they are reusable.

See it with a real tokenizer

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")        # downloads the tokenizer only
for word in ["unbelievable", "tokenization", "Surat"]:
    print(word, "->", tok.tokenize(word))
# unbelievable -> ['un', 'bel', 'iev', 'able']
# tokenization -> ['token', 'ization']
# Surat -> ['Sur', 'at']

“Tokenization” splits into two sensible pieces. “Unbelievable” splits into pieces that are useful for statistics but are not syllables, and a place name like “Surat” is cut because the tokenizer saw it rarely. The splits come from counting what was frequent in the training text, not from grammar.

Special tokens

Every tokenizer also reserves a few special tokens that mean something to the machinery: start and end of text, padding, and, for chat models, markers such as “user” and “assistant” that a chat template wraps around each message. To the model they are just more tokens.

Takeaways

• Characters: tiny vocabulary but very long sequences.
• Whole words: short sequences but a huge vocabulary, and unknown words.
• Subwords: common words stay whole, rare words are split into reusable pieces.
• Special tokens (end of text, padding, chat roles) are ordinary tokens with reserved meanings.

Next in the series (Part 2): Byte-pair encoding by hand: six merges