Before you rent a model, pretrain one
Build a Small GPT From Scratch · Part 1 of 8
Before you rent a model by the token, build the habit of training one. A small model on a free Colab GPU is enough, as long as you run the whole real loop and not a cartoon of it.
What “pretraining” means
Pretraining is the first and largest stage of making a language model: show it a huge pile of text and ask it, at every position, to predict the next token. Nothing else. No questions, no answers, no instructions. The result is a base model that has absorbed the patterns of its text.
The loop, in six steps
- Pack documents into fixed-length rows of tokens.
- Batch several rows together.
- Forward pass: the GPT turns every position into probabilities for the next token.
- Loss: compare those probabilities with the token that actually came next.
- Update: backpropagate and let the optimizer nudge the weights.
- Repeat, evaluating on held-out text now and then and saving checkpoints.
Why small is still real
A GPT of a few million parameters trained on a children’s-story corpus or a slice of Wikipedia text will not impress anyone with its answers. But it goes through exactly the decisions a large run does: how to feed data, how to set the learning rate, how to survive an interrupted session, when to stop. Doing it once at small scale makes every later claim about large models easier to judge.
The next seven posts take those decisions one at a time: packing, the learning-rate schedule, gradient accumulation, number formats and checkpoints, how to budget compute, what scaling laws say, and why the finished model is not yet a chatbot.
Takeaways
- Pretraining means predicting the next token over a very large body of text.
- The loop is: pack, batch, forward pass, loss, update, repeat, with evaluation and checkpoints.
- A small run uses the same decisions as a large one, which is why it is worth doing.
- The result is a base model, not an assistant.
Next in the series (Part 2): Packing documents: how not to waste compute on padding