Temperature, top-k and top-p: how models pick the next word

Tokenization & decoding · Part 7 of 8

If every run gave the identical answer, a chatbot would feel like a vending machine. Instead, models sample: they draw the next token at random in proportion to its probability. Three settings control how adventurous that draw is.

Temperature: sharpen or flatten

The model produces raw scores, called logits, for every token. Dividing them by a temperature before turning them into probabilities reshapes the whole distribution. Below 1 it sharpens it: the favourite gets even more of the probability. Above 1 it flattens it: unlikely tokens get a real chance. Take a toy next-word distribution for “The capital of Japan is…”:

• at temperature 0.5, “Tokyo” gets 96% of the probability;
• at temperature 1, 73%;
• at temperature 2, only 43%.

I drew 2,000 samples at each setting. At 0.5, 1920 of them were “Tokyo”. At 2, only 837 were, and nonsense such as “banana” appeared 40 times. Too low behaves like greedy decoding; too high becomes incoherent.

Top-k: only the k best

Top-k sampling throws away everything except the k most likely tokens, renormalises, and samples from what is left. It stops very unlikely tokens from ever being picked. Its weakness is that k is fixed. When the model is unsure and many tokens are nearly as likely, a small k cuts good options; when it is certain, a large k lets nonsense back in.

Top-p (nucleus): the smallest set that adds up to p

Top-p keeps the smallest set of tokens whose probabilities add up to at least p, such as 0.9. It adapts. When the model is confident, three tokens may already reach 0.9, so only three stay. When it is unsure, seven of eight may be needed. In the toy example above, top-k = 3 keeps only 45% of an uncertain distribution, while top-p = 0.9 keeps 7 of 8 tokens. Top-p is usually the best default.

The order matters

Temperature is applied to the scores first, then the cut-off, then the remaining probabilities are renormalised so they add to 1 before the draw.

import numpy as np

def softmax(z):
    z = z - z.max(); e = np.exp(z); return e / e.sum()

def sample_next(logits, temperature=1.0, top_k=None, top_p=None, rng=None):
    probs = softmax(np.asarray(logits, dtype=float) / temperature)       # temperature first
    order = np.argsort(-probs)
    keep = np.ones(len(probs), dtype=bool)
    if top_k is not None:
        keep[order[top_k:]] = False
    if top_p is not None:
        cutoff = int(np.searchsorted(np.cumsum(probs[order]), top_p) + 1)  # smallest set reaching top_p
        keep[order[cutoff:]] = False
    probs = np.where(keep, probs, 0.0); probs /= probs.sum()             # renormalise what is left
    return int(rng.choice(len(probs), p=probs))

Open in Colab Change the settings and watch the draws change. Free in your browser.

Takeaways

• Temperature sharpens (below 1) or flattens (above 1) the probabilities.
• Top-k keeps a fixed number of tokens; top-p keeps whatever number add up to p.
• Top-p adapts to how confident the model is, so it is usually the best default.
• Apply temperature first, cut, then renormalise before drawing.

Next in the series (Part 8): How a next-token predictor drafts a paragraph, and why it rambles