The one job of a language model: predict the next token

Tokenization & decoding · Part 4 of 8

Strip away the hype and a large language model has exactly one job: look at the text so far and predict the next token. Everything it can do is built on that single skill, trained on plain running text with no labels at all.

Free labels from ordinary text

Take any sentence and shift it by one token. The text so far is the input; the next token is the answer. No human has to annotate anything, because the text carries its own answers. That is why pretraining can use a huge amount of text.

At every position the model produces a probability for every token in the vocabulary. The loss rewards putting a high probability on the token that really came next:

loss = − Σt log P( xt | x< t )

Teacher forcing

During training the model is always shown the true previous tokens, not its own earlier guesses. This is called teacher forcing. It makes training stable and parallel, but it means the model is never trained on its own mistakes. That is one reason a low training loss does not by itself guarantee good generated text.

Perplexity: the effective number of choices

Average the loss over many tokens and raise e to that power, and you get perplexity. Read it as “the model is about as unsure as if it were choosing evenly among this many tokens.” A perplexity of 20 is like picking at random from 20 equally likely options. Smaller is better, and it is handy for comparing models. It does not measure whether the output is useful, which is why other kinds of evaluation exist.

import math

p_true = [0.50, 0.25, 0.10]      # probability the model gave to the real next token
loss = -sum(math.log(p) for p in p_true) / len(p_true)
print(round(loss, 3), round(math.exp(loss), 2))     # 1.461 4.31 : loss and perplexity

Three guesses of 0.5, 0.25 and 0.1 give an average loss of 1.46, a perplexity of 4.3. If the model had spread its probability evenly over 20 tokens every time, the perplexity would be exactly 20.

Takeaways

• Pretraining predicts the next token from the text so far; the text supplies its own labels.
• The loss is the negative log of the probability given to the true token.
• Teacher forcing trains on true history, so training loss is not the same as generation quality.
• Perplexity is e raised to the average loss: the effective number of equally likely choices.

Next in the series (Part 5): A language model made of nothing but counts