Neural networks in five minutes: the recap before the transformer

Transformer on-ramp · Part 1 of 8

Your team lead says: “Before we touch large language models, everyone must be able to read a transformer.” Half the room has never trained a neural network. This five-minute recap is the entry ticket — it is all the neural-network knowledge a transformer quietly assumes.

A neuron is a tiny weighted vote

A neuron takes a few numbers, multiplies each by a weight (how much it should matter), adds them up with a bias (a built-in nudge), and passes the result through a squashing function called the activation. In symbols: z = w·x + b, then a = σ(z). That is the whole idea. Everything bigger is made of copies of this.

Stack them and you get a network

Put many neurons side by side and you have a layer. Feed one layer’s outputs into the next and you have a network. The weights and biases are the network’s memory; they are called its parameters. A large language model is the same recipe with billions of them.

How a network learns: the four-step loop

1. Forward pass. Push an example through and get a prediction.
2. Loss. Measure how wrong the prediction was with a single number.
3. Backward pass. Use the chain rule (backpropagation) to work out, for every parameter, which way and how strongly it should move to reduce the loss. These are the gradients, and one backward sweep produces all of them.
4. Update. Nudge each parameter a small step against its gradient. That is gradient descent, and the step size is the learning rate.

Repeat that loop over lots of examples and the loss falls. Nothing else is happening.

One real step, in a few lines

Here is a single neuron taking one learning step toward a target of 1. The numbers are small enough to check by hand.

import numpy as np

x = np.array([0.5, -1.0, 2.0])
w = np.array([0.4,  0.3, -0.2]); b = 0.1
sigmoid = lambda z: 1 / (1 + np.exp(-z))

a = sigmoid(w @ x + b)            # forward: a = 0.401
loss = 0.5 * (a - 1.0) ** 2       # loss  = 0.179

dz = (a - 1.0) * a * (1 - a)      # backward: chain rule
w -= 0.5 * dz * x; b -= 0.5 * dz  # update, learning rate 0.5

a2 = sigmoid(w @ x + b)           # forward again: a = 0.512
loss2 = 0.5 * (a2 - 1.0) ** 2     # loss  = 0.119

One step moved the prediction from 0.401 towards the target and cut the loss from 0.179 to 0.119. A real network does this for millions of parameters at once, but the logic is identical.

Takeaways

• A neuron is a = σ(w·x + b): weigh, add, squash.
• Layers stack neurons; the weights and biases are the parameters.
• Learning is a loop: forward, loss, backward, update.
• Backpropagation gives every parameter’s gradient in one sweep.

Next in the series (Part 2): how text becomes numbers — tokens turning into vectors.