Packing documents: how not to waste compute on padding
Documents come in all lengths, but a GPU wants rectangles. How you turn one into the other decides how much of your compute is spent on real text and how much on nothing.
READ MORE
Before you rent a model, pretrain one
Before you rent a model by the token, build the habit of training one. A small model on a free Colab GPU is enough, as long as you run the whole real loop and not a carto...
READ MORE
How a next-token predictor drafts a paragraph, and why it rambles
A skeptical manager asks: “Explain, in terms our editors can follow, how a model that only predicts the next token can draft a paragraph, and why it sometimes rambles.” A...
READ MORE
Temperature, top-k and top-p: how models pick the next word
If every run gave the identical answer, a chatbot would feel like a vending machine. Instead, models sample: they draw the next token at random in proportion to its proba...
READ MORE
Greedy and beam search: the best word or the best sentence?
A model gives you probabilities for the next word. Turning those into text means choosing. The obvious choice, always take the most likely word, has a flaw that a tiny ex...
READ MORE
A language model made of nothing but counts
Before neural networks, language models were just tables of counts. Build one and the ideas from the last post, probabilities, perplexity and sampling, stop being abstrac...
READ MORE