Base model or chat model? What your tiny GPT can and cannot do
When your small GPT finishes training, it will not answer a question. It will do something that looks odd until you remember what it was asked to do all along.
READ MORE
Chinchilla, and why recent models are over-trained on purpose
For a fixed budget, should you train a bigger model on less data or a smaller model on more? A famous study gave a clear answer, and then the industry deliberately ignore...
READ MORE
How much will a training run cost? C = 6ND
Before you spend a GPU-hour, you can estimate what a training run will cost with one line of arithmetic. It is rough, but it is accurate enough to stop you from planning...
READ MORE
Half the bytes: mixed precision and checkpoints
Two habits separate a run that finishes from one that does not: storing numbers in fewer bytes, and saving progress often enough that an interruption costs minutes instea...
READ MORE
Gradient accumulation: a big batch on a small GPU
Large batches make training steadier, but a free GPU can only hold a small one. Gradient accumulation closes the gap: run several small batches, add up their gradients, a...
READ MORE
Warmup then cosine: the learning-rate shape of big training runs
Almost every large training run starts the learning rate near zero, climbs to a peak, then slowly brings it back down. Here is what that shape looks like with real number...
READ MORE