Positional encoding: teaching attention about word order

Transformer on-ramp · Part 7 of 8

Attention treats a sentence like a bag of words: shuffle them and nothing changes. Yet “the dog bit the man” and “the man bit the dog” mean very different things. Position has to be put back in by hand.

Attention cannot see order

Look at the machinery from the last three posts. Nothing in it knows which token came first. Shuffle the tokens and every score is still computed from the same queries and keys, just in a different order. In an experiment, I fed a self-attention layer the same six tokens in two different orders: the output for each token was identical, merely shuffled along with the input. As far as attention is concerned, “dog bites man” and “man bites dog” are the same bag of words.

The fix: add position information to the token vectors before attention sees them.

The original recipe: a pattern of waves

The first transformer paper added a fixed pattern to each token vector. For position pos and dimension pair i, the pattern is a sine and a cosine at a frequency that depends on i:

PE(pos, 2i) = sin(pos / 100002i/d)   PE(pos, 2i+1) = cos(pos / 100002i/d)

Early dimensions oscillate quickly and later ones slowly, a bit like the hands of a clock. Every position gets its own combination of readings, so the model can tell positions apart.

Making the pattern

import torch

def sinusoidal_positions(T, d):
    pos = torch.arange(T).unsqueeze(1)           # 0 … T-1
    i = torch.arange(0, d, 2)                    # even dimensions
    pe = torch.zeros(T, d)
    pe[:, 0::2] = torch.sin(pos / 10000 ** (i / d))
    pe[:, 1::2] = torch.cos(pos / 10000 ** (i / d))
    return pe

pe = sinusoidal_positions(50, 16)
print(pe.shape)            # torch.Size([50, 16])
print(pe[0, :4])           # tensor([0., 1., 0., 1.])   position 0
print(pe[1, :4])           # tensor([0.8415, 0.5403, 0.3110, 0.9504])   position 1
# x = token_embeddings + pe[:T]   <- this sum is what the first attention layer sees

What modern models do

Many models simply learn a position vector per position instead. Newer ones use RoPE, which rotates the query and key by an angle that grows with position, so attention itself depends on how far apart two tokens are. That is covered in a later chapter of the course.

Takeaways

• Self-attention on its own is blind to word order.
• Fix: add a position signal to each token vector before attention.
• The original design uses sine and cosine waves at different speeds.
• Learned positions and RoPE are common in modern models.

Next in the series (Part 8): The transformer block: putting the pieces together