Multi-head attention: many ways of looking at one sentence
If one attention head gives a sentence one point of view, why not give it several? That is the whole idea of multi-head attention, and it turns out to be almost free.
READ MORE
Self-attention from scratch in PyTorch
Reading about attention is one thing. Typing it is another. Let us build it from scratch in PyTorch, in about ten lines, and then check it against the library.
READ MORE
Self-attention: every word looks at every other word
A transformer reads a sentence by letting every word ask every other word one question: “how much should I pay attention to you?” That one idea, self-attention, is what s...
READ MORE
Why recurrence isn't enough: reading a sentence all at once
Before transformers, language models read like a person reading with a finger: one word after another. That works until the sentence gets long, or until you want to train...
READ MORE
From words to numbers: how tokens become vectors
A language model cannot read the word “bank”. All it can do is arithmetic. So the very first job in any LLM is to turn text into numbers, and to do it in a way that keeps...
READ MORE
Neural networks in five minutes: the recap before the transformer
Your team lead says: “Before we touch large language models, everyone must be able to read a transformer.” Half the room has never trained a neural network. This five-min...
READ MORE