Not every language pays the same price

Tokenization & decoding · Part 3 of 8

Here is a sentence a delivery company might send: “The invoice will be paid next Friday.” Same meaning, five languages. How many tokens does each one cost?

Measure it

I counted tokens for that sentence and four translations with three real, widely used tokenizers. The brackets show the count relative to English for the same tokenizer:

Why does this happen?

A tokenizer’s merges are learned from its training text. A tokenizer built from mostly English text turns English words into one or two tokens, and text in other scripts into many small pieces, sometimes down to individual bytes. The newer Qwen tokenizer, with a bigger and more multilingual vocabulary, cuts the worst gap here from 6.6 times to 4.0 times. XLM-RoBERTa, trained on text in about a hundred languages, is nearly flat on these five sentences.

Why you should care: three bills

1. Money. Hosted models charge per token, so the identical answer can cost several times more in one language than in another.
2. Context. A model can only look at a fixed number of tokens at once. If your language needs more tokens, less of your document fits.
3. Quality. When one word is spread over many tiny pieces, each piece carries little meaning and the model has to put the word back together, often in a language it saw less of in training.

A 2023 study by Petrov and colleagues measured the same content needing up to fifteen times as many tokens in some languages as in others. Five sentences prove nothing general, and the study found large gaps even in tokenizers built for many languages. The safe advice is simple: measure your own text with the tokenizer of the model you plan to use.

from transformers import AutoTokenizer

sentences = {"English": "The invoice will be paid next Friday.",
             "Hindi":   "??? ???? ???????? ?? ?????? ??????"}
for name in ["gpt2", "Qwen/Qwen2.5-0.5B"]:
    tok = AutoTokenizer.from_pretrained(name)
    print(name, {lang: len(tok.encode(s, add_special_tokens=False)) for lang, s in sentences.items()})
# gpt2 {'English': 8, 'Hindi': 53}
# Qwen/Qwen2.5-0.5B {'English': 8, 'Hindi': 32}

Takeaways

• The same meaning can need very different numbers of tokens in different languages.
• In this test GPT-2 needed 53 tokens for Hindi against 8 for English.
• More tokens means more money, less usable context and often lower quality.
• Always measure your own text with the tokenizer of the model you will use.

Next in the series (Part 4): The one job of a language model: predict the next token