Why language models read tokens, not words
Hand a model the word “unbelievable” and ask it to read. What should it look at: the letters, the whole word, or something in between? That choice, made before any learni...
READ MORE
Byte-pair encoding by hand: six merges
Where do those subword pieces come from? From an almost embarrassingly simple procedure called byte-pair encoding, or BPE: start with letters, and keep gluing together th...
READ MORE
Not every language pays the same price
Here is a sentence a delivery company might send: “The invoice will be paid next Friday.” Same meaning, five languages. How many tokens does each one cost?
READ MORE
The one job of a language model: predict the next token
Strip away the hype and a large language model has exactly one job: look at the text so far and predict the next token. Everything it can do is built on that single skill...
READ MORE
A language model made of nothing but counts
Before neural networks, language models were just tables of counts. Build one and the ideas from the last post, probabilities, perplexity and sampling, stop being abstrac...
READ MORE
Greedy and beam search: the best word or the best sentence?
A model gives you probabilities for the next word. Turning those into text means choosing. The obvious choice, always take the most likely word, has a flaw that a tiny ex...
READ MORE
Temperature, top-k and top-p: how models pick the next word
If every run gave the identical answer, a chatbot would feel like a vending machine. Instead, models sample: they draw the next token at random in proportion to its proba...
READ MORE
How a next-token predictor drafts a paragraph, and why it rambles
A skeptical manager asks: “Explain, in terms our editors can follow, how a model that only predicts the next token can draft a paragraph, and why it sometimes rambles.” A...
READ MORE