How a next-token predictor drafts a paragraph, and why it rambles

Tokenization & decoding · Part 8 of 8

A skeptical manager asks: “Explain, in terms our editors can follow, how a model that only predicts the next token can draft a paragraph, and why it sometimes rambles.” After seven posts, you can answer exactly that.

The loop

Drafting a paragraph is one tiny step repeated hundreds of times:

1. Turn the prompt into tokens.
2. Run the model once: it gives a probability for every possible next token.
3. Pick one with the decoding rule (greedy, beam, or sampling with temperature, top-k and top-p).
4. Append it to the text and go back to step 2.
5. Stop at the end-of-text token or at a length limit.

Each token is chosen knowing everything written so far, which is why the text is usually fluent and consistent from word to word. The model learned fluent continuations from an enormous amount of text.

Why it sometimes rambles

• No plan. The model does not decide where the paragraph is going. It only decides the next token, so it can drift.
• Mistakes are never undone. A poor token early on becomes part of the context for everything after it, and greedy decoding cannot go back.
• Greedy loops. Always taking the most likely word can fall into repeating itself. Our count-table model shows a version of this: with a long context it copies its training text, as in “The warehouse opens at the works together to worry. At night the start”.
• Too much heat. At a high temperature unlikely tokens get picked and the text wanders into nonsense.
• Training is not generation. With teacher forcing, the model is trained on true history but generates from its own, so small slips can compound.

What an editor can do

Lower the temperature or use top-p around 0.9 for steadier text; set a length limit and a sensible stop; and remember that the model writes fluent text because it has seen a lot of it, not because it has checked the facts. The next time a draft rambles, you will know which of the five causes to suspect.

Takeaways

• Generation is one loop: tokens in, probabilities out, pick one, append, repeat.
• There is no plan, only the next token, which explains both fluency and drifting.
• Rambling comes from drift, uncorrected mistakes, greedy loops, high temperature and the training/generation gap.
• Decoding settings are the main control you have over how adventurous the text is.

That completes the series. The chapter in the course goes further, with a full hands-on lab, a self-test and optional deep-dives.