Before you rent a model, pretrain one
Before you rent a model by the token, build the habit of training one. A small model on a free Colab GPU is enough, as long as you run the whole real loop and not a carto...
READ MORE
Packing documents: how not to waste compute on padding
Documents come in all lengths, but a GPU wants rectangles. How you turn one into the other decides how much of your compute is spent on real text and how much on nothing.
READ MORE
Warmup then cosine: the learning-rate shape of big training runs
Almost every large training run starts the learning rate near zero, climbs to a peak, then slowly brings it back down. Here is what that shape looks like with real number...
READ MORE
Gradient accumulation: a big batch on a small GPU
Large batches make training steadier, but a free GPU can only hold a small one. Gradient accumulation closes the gap: run several small batches, add up their gradients, a...
READ MORE
Half the bytes: mixed precision and checkpoints
Two habits separate a run that finishes from one that does not: storing numbers in fewer bytes, and saving progress often enough that an interruption costs minutes instea...
READ MORE
How much will a training run cost? C = 6ND
Before you spend a GPU-hour, you can estimate what a training run will cost with one line of arithmetic. It is rough, but it is accurate enough to stop you from planning...
READ MORE
Chinchilla, and why recent models are over-trained on purpose
For a fixed budget, should you train a bigger model on less data or a smaller model on more? A famous study gave a clear answer, and then the industry deliberately ignore...
READ MORE
Base model or chat model? What your tiny GPT can and cannot do
When your small GPT finishes training, it will not answer a question. It will do something that looks odd until you remember what it was asked to do all along.
READ MORE