The one job of a language model: predict the next token
Strip away the hype and a large language model has exactly one job: look at the text so far and predict the next token. Everything it can do is built on that single skill...
READ MORE
Not every language pays the same price
Here is a sentence a delivery company might send: “The invoice will be paid next Friday.” Same meaning, five languages. How many tokens does each one cost?
READ MORE
Byte-pair encoding by hand: six merges
Where do those subword pieces come from? From an almost embarrassingly simple procedure called byte-pair encoding, or BPE: start with letters, and keep gluing together th...
READ MORE
Why language models read tokens, not words
Hand a model the word “unbelievable” and ask it to read. What should it look at: the letters, the whole word, or something in between? That choice, made before any learni...
READ MORE
The transformer block: putting the pieces together
Now we put the pieces together. A transformer is not a mystery: it is one small block repeated many times. If you followed Parts 1 to 7, you already know every part of it...
READ MORE
Positional encoding: teaching attention about word order
Attention treats a sentence like a bag of words: shuffle them and nothing changes. Yet “the dog bit the man” and “the man bit the dog” mean very different things. Positio...
READ MORE