Why recurrence isn't enough: reading a sentence all at once
Transformer on-ramp · Part 3 of 8
Before transformers, language models read like a person reading with a finger: one word after another. That works until the sentence gets long, or until you want to train on a mountain of text.
How a recurrent network reads
A recurrent network (RNN) reads a sentence one token at a time. After each token it updates a small memory, the hidden state, and hands it to the next step. Gated versions such as the LSTM are better at deciding what to keep, but the basic shape is the same: a chain.
Two problems with a chain
- It is slow to train. Step 5 needs the result of step 4, so you cannot process all positions at once. Modern GPUs are built to do many things in parallel, and a chain wastes that.
- Long-range links fade. Take “The keys that I left there are missing.” The verb has to agree with “keys”, which is far away. Everything in between has to carry that fact forward, and even with gating it tends to degrade over long distances.
The transformer’s answer
Throw out the chain. Let every token look at every other token directly, all at once. The path between any two words is one hop, however far apart they are, and the whole sentence can be processed in parallel. That mechanism is called attention, and it is the subject of the next three parts.
Takeaways
- A recurrent network is a chain: one hand-off per token.
- Chains train slowly because steps cannot run in parallel.
- Information from far back fades, even with gating.
- Attention connects every token to every other token in one hop, at a cost that grows with the square of the length.
Next in the series (Part 4): Self-attention: every word looks at every other word