Base model or chat model? What your tiny GPT can and cannot do

Build a Small GPT From Scratch · Part 8 of 8

When your small GPT finishes training, it will not answer a question. It will do something that looks odd until you remember what it was asked to do all along.

A base model continues text

The only thing the next-token objective ever asked of the model was: given this text so far, make the real next token likely. A question in the middle of such text is just more text. A base model given “What is the capital of France?” may well answer with a second question, or a list of similar questions, because that is what often follows a question in a document. It is not being difficult; nothing in its training said that questions deserve answers.

What turns it into an assistant

The assistant behaviour, following instructions in a helpful conversational voice, comes from further training on examples of instructions and good replies, and often a preference-based stage after that. That is fine-tuning, covered in a later chapter of the course. Keep the two stages separate in your head: pretraining gives the model knowledge of language and the world, fine-tuning gives it manners.

How to read your own run

A practical way to judge a pretraining run is to look at its loss curve and a gallery of generated samples side by side. Three common patterns, each with a different fix:

• The loss spikes or climbs. The learning rate is probably too high or the warmup too short.
• The loss is still falling when the run stops. The model is undertrained: it needs more steps or more tokens, not a bigger design.
• The loss looks fine but the samples are repetitive or odd. Suspect the data: duplicates, boilerplate or noise.

These are rules of thumb, not a diagnosis, but they tell you where to look first.

Takeaways

• A base model continues text; it does not follow instructions.
• Instruction-following comes from fine-tuning, not from pretraining.
• Read a run by its loss curve and its sample gallery together.
• Rising loss, a curve still falling at the end, and odd samples each point to a different fix.

That completes the series. The chapter in the course goes further, with a full hands-on lab, a self-test and optional deep-dives.