From words to numbers: how tokens become vectors

Transformer on-ramp · Part 2 of 8

A language model cannot read the word “bank”. All it can do is arithmetic. So the very first job in any LLM is to turn text into numbers, and to do it in a way that keeps meaning.

Step 1: cut the text into tokens

The text is first chopped into tokens: small pieces that are often whole words and sometimes parts of words. Each token has a fixed whole-number ID in the model’s vocabulary. (How the cutting is decided is the subject of the next chapter of the course.) An ID is only a label: token 5 is not “more” or “closer to” token 6 in any sense.

Step 2: look up a vector

Labels cannot carry meaning, so each ID is swapped for a list of numbers. The model keeps an embedding table: one row per vocabulary item, with the same number of columns for every row. The token ID is simply the row number, and the row is that token’s vector. There is no calculation here, just a lookup.

Where do the numbers come from?

They start out random and are learned together with everything else. If two tokens tend to appear in similar contexts, training nudges their rows closer together, because that helps the model predict text. Nobody hand-designs the numbers; whatever similarity the model finds falls out of training.

Why numbers that live in space are useful

Once a token is a point in space, the model can reason with distance and direction instead of arbitrary labels. Layers after the table are ordinary neural-network layers, and they can only work on numbers like these.

The whole idea in four lines

import torch

emb = torch.nn.Embedding(num_embeddings=8, embedding_dim=4)   # 8 tokens, 4 numbers each
ids = torch.tensor([[2, 5, 7]])         # "The cat sat" as token IDs
vectors = emb(ids)                      # look up three rows
print(vectors.shape)                    # torch.Size([1, 3, 4]): batch, tokens, numbers

Three token IDs went in; three 4-number vectors came out. A real model does exactly this with a much larger table.

Takeaways

• Text is cut into tokens, and each token has an integer ID.
• An embedding table is a lookup: ID in, vector out.
• The table’s numbers are learned during training, not designed by hand.
• Vectors let the model use distance and direction instead of labels.

Next in the series (Part 3): Why recurrence isn't enough: reading a sentence all at once