2. The math of learning: error, slope and gradient – How LLMs work

The network has knobs. How do we set them?

In lecture 1 you saw a network that reads handwritten digits. It has 13,002 adjustable numbers: weights and biases. When they are set well, the network answers correctly. When they are random, the network guesses blindly.

So learning means one thing: finding a good setting of the knobs. Nobody tunes them by hand. The computer does it by itself, step by step. To do that it needs two things:

A number that says how badly the network is doing. It is called the cost (or loss). The smaller, the better.
A way to find out which way to turn each knob so the cost goes down. That is what the slope, i.e. the derivative, is for, and with many knobs, the gradient.

That is all the math you need to understand how neural networks learn, even the largest ones. We won't prove anything. We will show what each idea means, and you will try all of it.

One example through the whole lecture

Instead of 13,002 knobs we start with one. An ice-cream stand sells cones: 1 scoop costs €2.50, 2 scoops €4.00 and 3 scoops €5.50. Our “model” should guess the price from the number of scoops. With this example we will build the cost, the slope and the gradient. At the end we add a second neuron and get to the chain rule.

Refresher: a function is a number machine

A function is a machine: you put a number in, another number comes out. The same input always gives the same output. We write it like this:

price = w · x x is the number of scoops (input), w is the price per scoop (the knob, a weight), price is the output.

If you set w = 2, the machine turns 3 scoops into a price of 2 · 3 = 6 euros. This is exactly one neuron from lecture 1: one input, one weight, for now without a bias and without an activation function.

A graph is a picture of a function. The input goes on the horizontal axis, the output on the vertical one, and you draw a point for every input. The function w · x is a straight line through zero. The bigger w, the steeper the line.

scoops x	real price y	guess with w = 1	guess with w = 2
1	€2.50	€1.00	€2.00
2	€4.00	€2.00	€4.00
3	€5.50	€3.00	€6.00

With w = 1 the model underestimates every price. With w = 2 it is much better, but still not exact. Which w is best? For that we need a single number that measures “bad”.

Two kinds of numbers

Notice the difference. x and y are data: they are given and we don't change them. w is a parameter: that is what we tune. During learning we keep asking what happens when we move a parameter, while the data stay fixed.

The cost: one number that says “how bad”

For each example we know what the model guessed and what the truth is. Their difference is the error. With w = 1 and one scoop the error is 1.00 − 2.50 = −1.50.

Errors can be positive or negative, and if we added them up they would cancel out. So we square each one. A square is never negative and it punishes big errors much more than small ones: an error of 2 gives 4, an error of 0.5 only 0.25. Finally we average the squares over all examples:

C(w) = [ (w·1 − 2.5)² + (w·2 − 4)² + (w·3 − 5.5)² ] / 3 C is the cost. The average of the squared errors is called the mean squared error.

With w = 1 we get (2.25 + 4 + 6.25) / 3 ≈ 4.167. With w = 2 we get (0.25 + 0 + 0.25) / 3 ≈ 0.167. Smaller number, better model. You can even draw the cost literally: an error is the side of a square, and the cost is the average area of the squares.

Try it

Find the price per scoop with the lowest cost

Move w with the slider or drag directly in the plots. On the left is the model's line and the red squares of the errors. On the right is the cost C for every possible w, and the orange dot is your w.

Task: find the w with the lowest cost. You'll end up around w ≈ 1.93 with a cost of about 0.143. Notice that you can never reach zero: a line through zero cannot hit all three points at once.

The plot on the right is the key picture of this lecture. The cost is a function too. Its input isn't scoops but the parameter w. For every setting of the knob it tells you how badly the model works. Learning means finding the lowest point of this curve.

Why can't the cost reach zero? Because the real price is €1 for the cone + €1.50 per scoop. The cone costs something even with no scoops, and a line through zero can't express that. We need a second knob, the bias b: price = w · x + b. We'll come back to it in chapter 7.

How it works for the digit network

The network from lecture 1 has 10 outputs. For a picture of a three we want output “3” to be 1 and the others 0. The cost of one picture is the sum of 10 squared errors. The total cost is the average over all 60,000 training pictures. Same principle, just more numbers. And the cost is a function of all 13,002 parameters at once.

Slope: what happens when you nudge a knob

With one knob we could find the best w by trying a thousand values. With 13,002 knobs that's impossible, there are more combinations than atoms in the universe. We need a smarter question: if I nudge a knob a little, does the cost go up or down? And by how much?

The answer is the slope of the curve at that spot. You know it from roads: a 10% grade means that for 100 meters forward you climb 10 meters. Slope is “how much up” divided by “how much forward”:

slope ≈ Δf / Δx = [ f(x + Δx) − f(x) ] / Δx Δ (delta) means “change”. Δx is a small nudge of the input, Δf is the change of the output it causes.

On a curve the slope changes from place to place. So we take the smallest possible nudge. As Δx approaches zero, the ratio Δf / Δx settles on one number. That number is the derivative of the function at that point. It is written f′(x) or df/dx. In the picture it is the slope of the tangent: the line that just touches the curve at that point.

Try it

Nudge a function and measure its slope

Drag in the plot to pick a point x. The blue line goes through x and through the point Δx further. The orange one is the tangent, the exact slope. Shrink Δx and watch the blue line approach the orange one.

Task: for f(x) = x² measure the slope at 1, 2 and −1. Can you guess the rule? (It is always 2·x.) Then switch to “hill and valley” and find the places where the slope is zero. They are the tops of hills and the bottoms of valleys.

A derivative carries two pieces of information and we will need both:

The sign tells the direction. Positive slope: make x bigger and f goes up. Negative slope: make x bigger and f goes down.
The size tells the sensitivity. A slope of 8 means a nudge of 0.01 changes the output by about 0.08. A slope of 0.1 means the output barely moves.

Key idea

The derivative answers one question: if I nudge the input a little, how many times more does the output move? Change of output ≈ derivative × change of input. This one sentence is enough for the rest of the lecture.

Five rules you actually need

Derivatives don't have to be measured by nudging. For common functions there are ready-made rules. We won't prove them, just show what they say and why they make sense. For neural networks this handful is enough:

function	derivative	why it makes sense
c (constant)	0	A number that doesn't change won't change when you move x. A flat line has zero slope.
a · x	a	A straight line has the same slope everywhere. For example `w · x` has slope x with respect to w: raise the price per scoop by 1 cent and 3 scoops cost 3 cents more.
x²	2 · x	A parabola is flat at zero and gets steeper the further you go. You checked it in the demo above.
c · f(x)	c · f′(x)	If a function is three times taller, its climb is three times steeper.
f(x) + g(x)	f′(x) + g′(x)	When heights add up, climbs add up. That is why we can differentiate the cost example by example and average the results.
σ(x) sigmoid	σ(x) · (1 − σ(x))	The slope of the “S” can be computed from its own value. It is largest in the middle (0.5 · 0.5 = 0.25) and almost zero at the edges, where σ is close to 0 or 1.

Let's try it on our ice cream. If you multiply out the brackets in the cost formula and collect equal terms, you get an ordinary parabola:

C(w) = 4.667 · w² − 18 · w + 17.5 → C′(w) = 9.333 · w − 18 Rules for x², c·f, a·x, constants and sums. The number 4.667 is 14/3, so the derivative is 2 · 14/3 = 9.333.

At w = 1 the slope is 9.333 − 18 = −8.67. It is negative, so making w bigger lowers the cost. At w = 2 the slope is +0.67, so we overshot a little. And the slope is zero at w = 18 / 9.333 ≈ 1.93. That is exactly the bottom you found in the first demo.

Why not compute the bottom directly?

For a parabola you can, just set the derivative to zero. For a network with thousands of parameters and sigmoids, such equations can't be solved. But we can always compute the slope where we currently are. And that is enough to reach the bottom in steps.

Gradient descent: a ball rolling downhill

Picture the cost curve as a valley with a ball on it. The ball can't see the whole valley, it only feels the slope beneath it. If the ground rises to the right, it rolls left. If it rises to the left, it rolls right. And on a steep slope it moves more than on a flat one.

That is exactly what the gradient descent algorithm does. It repeats a single step:

w_new = w − η · C′(w) η (eta) is the learning rate: how big a step we take. The minus means we go against the slope, i.e. downhill.

Example: we are at w = 1, the slope is −8.67 and the learning rate is η = 0.05. The new w = 1 − 0.05 · (−8.67) = 1 + 0.43 = 1.43. The negative slope moved us right, towards the bottom. Near the bottom the slope is small, so the steps shrink by themselves.

Try it

Let the ball roll down the cost curve

The ball sits on the cost curve of the ice-cream example. The orange arrow shows its next step. The table under the plot lists the numbers of every step. Drag in the plot to change the start.

Task: try learning rates 0.02 (slow crawl), 0.10 (almost one jump), 0.20 (jumps from side to side) and 0.25 (flies off). Then switch to the “bumpy landscape” and start once to the right and once to the left of the little hill. You don't always end up in the deepest valley.

The learning rate is one of the most important settings when training any network:

small ηSafe but slow

Every step is reliable, but you need thousands of them. For a big network that means days of extra compute.

good ηQuickly to the bottom

A few steps and we are there. A good value can't be computed in advance, it is found by trying.

large ηOvershooting

The step jumps over the bottom to the other side, even higher. The cost grows instead of shrinking.

A bit of history

The French mathematician Augustin-Louis Cauchy described the method of descending along the slope as early as 1847, when he was solving systems of equations in astronomy. Neural networks use it almost unchanged.

Two knobs: partial derivatives and the gradient

Let's add a bias so the model can charge for the cone too: price = w · x + b. Now the cost has two inputs, w and b. It is no longer a curve but a landscape: each pair (w, b) has its own “altitude” C. We draw it from above like a map, where darker means lower cost and the lines join places of equal cost, like contour lines on a hiking map.

How do you find the slope in a landscape? Simply: move only one knob at a time and keep the other fixed. That gives two slopes, called partial derivatives:

∂C/∂w = 2 · ( 4.667·w + 2·b − 9 ) ∂C/∂b = 2 · ( 2·w + b − 4 ) The sign ∂ reads “partial”. ∂C/∂w: how much the cost changes when you move only w and keep b fixed.

Put both numbers together into a pair. That pair is the gradient, written ∇C (“nabla C”). It can be drawn as an arrow on the map, and it has a beautiful property: it points in the direction where the cost rises most steeply. We want to go down, so we go exactly the opposite way, in the direction −∇C. The gradient descent step is the same as before, just for each knob separately:

w ← w − η · ∂C/∂w b ← b − η · ∂C/∂b

Try it

The cost map and descent in two dimensions

The map shows the cost for all pairs (w, b). Click or drag on the map to pick a start. The white arrow is the direction −∇C, where the next step goes. Top right you see the line that this (w, b) draws through the data.

Task: run the descent and watch the line on the right settle onto the points. At the bottom you'll have w = 1.50 and b = 1.00, exactly the real price list. Notice that the trail first drops quickly into a long narrow trough and then crawls slowly along it. With a learning rate of 0.19 or more, the descent starts oscillating and flies off the map.

Why doesn't the trail head straight to the target? Because the gradient points in the steepest direction where you stand right now, not towards the target. In a narrow trough the steepest way is across it, not along it. Descent is local and short-sighted, but reliable.

From two knobs to 13,002

A landscape with two knobs can be drawn. The digit network has 13,002 knobs and its cost is a landscape in 13,002 dimensions. Nobody can draw that. But the recipe doesn't change by a single letter:

For each of the 13,002 parameters, compute the partial derivative of the cost.
Those 13,002 numbers form the gradient, an arrow in 13,002-dimensional space.
Move each parameter by −η times its number in the gradient. Repeat.

The gradient tells you which knob matters

Each number in the gradient means the same as the slope in one dimension. The sign tells whether to make the parameter bigger or smaller. The size tells how sensitive the cost is to it. If one weight has 3.2 and another 0.01, the first one matters about 300 times more for the cost. So the gradient is also a ranking: which changes give the most for the least.

∇C = ( ∂C/∂w₁, ∂C/∂w₂, …, ∂C/∂w_13,002 ) For example (+0.18, −3.20, +0.01, …): the second weight should go up a lot, the third barely matters.

Local minima

In the bumpy landscape from the chapter 6 demo you saw that the ball ends up in the nearest valley, not necessarily the deepest. Such a valley is called a local minimum. Gradient descent has no guarantee that it finds the best possible setting.

Surprisingly, for big networks this doesn't matter that much. There is a huge number of valleys and most are almost equally good. And in thousands of dimensions it is hard to get stuck: even if one direction is a bottom, among the thousands of other directions there is usually still a way down.

Key idea

Training a neural network = gradient descent on the cost function. The only hard part is computing the gradient, i.e. 13,002 partial derivatives (175 billion for GPT-3). That is what the chain rule, coming up now, and the backpropagation algorithm from the next lecture are for.

The chain rule: nudges multiply

In a network the cost doesn't depend on a weight directly. The weight affects a weighted sum, that affects a neuron's activation, the activation affects the next neuron, and only that one affects the cost. It is a chain. How do you find out how much the cost depends on a weight at the start of the chain?

Think of gears. The first gear turns the second three times faster, the second turns the third twice as fast. How much faster does the third turn than the first? 3 · 2 = 6 times. That is exactly the chain rule: the local slopes of the links multiply.

Let's build the smallest possible “deep” network: two neurons in a row, each with one weight and a sigmoid. Make it a toy from the ice-cream stand: the input is x = 2 scoops, the output should say whether the customer will be happy, and the target is y = 1 (they were). The computation goes like this:

z₁ = w₁·x → a₁ = σ(z₁) → z₂ = w₂·a₁ → a₂ = σ(z₂) → C = (a₂ − y)² Each arrow is a simple function and its derivative is in the table from chapter 5.

How much does the cost depend on the first weight? Multiply the slopes of all the links in the chain:

∂C/∂w₁ = ∂z₁/∂w₁ · ∂a₁/∂z₁ · ∂z₂/∂a₁ · ∂a₂/∂z₂ · ∂C/∂a₂ = x · a₁(1−a₁) · w₂ · a₂(1−a₂) · 2(a₂−y) The notation looks like fractions that “cancel”: ∂z₁ top and bottom, ∂a₁ top and bottom… It is not a proof, but a good way to remember it.

Try it

A chain of two neurons: multiply the slopes and check by nudging

Move the weights w₁ and w₂. The chain shows the value of every link and, between them, the local derivatives. Their product is ∂C/∂w₁. The “Nudge” button changes w₁ by 0.01 and compares the real change in cost with the prediction.

Task: press “Nudge w₁” and compare the two numbers. They almost match: change in cost ≈ derivative × 0.01. Then press “Learning step” a few times and watch the cost drop. Finally set w₁ = 4: the sigmoid is saturated, a₁(1−a₁) is almost zero and the whole product goes dark. The first weight then barely learns.

The last point of the task matters. When any link of the chain is almost zero, the whole product is almost zero. In deep networks with sigmoids the slope fades away towards the first layers. This is called the vanishing gradient, and it is one of the reasons today's networks prefer ReLU.

Just like in the digit network

In a real network each neuron has many inputs and a bias: z = w·a + … + b. The chain branches, but the rule doesn't change. The derivative with respect to the bias is even simpler, because ∂z/∂b = 1. This is exactly the computation from the 3Blue1Brown video Backpropagation calculus.

Stochastic gradient descent: fast and a little drunk

The cost is an average over all training examples. To take one honest step we would have to go through all 60,000 digit pictures. For language models it is hundreds of billions of words. That's far too slow.

The trick: shuffle the data randomly and split it into small piles, say 100 examples each. A pile is called a mini-batch. We compute the gradient from just one mini-batch and take a step right away. It is only an estimate of the real gradient, a bit imprecise, but it costs a fraction of the compute. This is called stochastic gradient descent (SGD, “stochastic” means random). Once we have gone through all the data like this, one epoch has passed.

3Blue1Brown compares it to the difference between a careful man who measures the slope precisely before every step and a drunk man who stumbles, but takes many more steps. Going downhill, the drunk man almost always gets there first.

Try it

A race: all the data against mini-batches

The model price = w·x + b learns from 12 purchases. The teal runner always computes the gradient from all 12 examples. The orange one estimates it from a mini-batch and walks on right away. Both process the same number of examples. Click the map to change the start.

Task: start the race with a mini-batch of 1 and see who gets to the green cross first. The orange trail is wobbly, but for the same amount of work it gets further. Then try a mini-batch of 6. The trail is calmer, but there are fewer steps.

mini-batchA pile of examples

One gradient estimate and one step come from it. Typically tens to thousands of examples.

epochOne pass through the data

The digit network is trained for tens of epochs. Large language models often see most of their data only once.

3.2 MGPT-3's batch

The largest GPT-3 learned from mini-batches of 3.2 million tokens with a learning rate of 0.00006 (Brown et al., 2020).

Mini exam

Eight questions, some of them need calculating. A calculator is allowed, paper is recommended. Nothing is sent anywhere.

1. The model is price = 2 · x. Data: 1 scoop for €2.50 and 2 scoops for €4.00. What is the cost C (the average of squared errors)?

2. What is the slope (derivative) of f(x) = x² at x = 3?

3. Parameter w = 3, slope of the cost C′(w) = 4, learning rate η = 0.1. What is w after one step of gradient descent?

4. The slope of the cost with respect to a weight came out negative. What does gradient descent do?

5. The gradient of the cost for two parameters is (∂C/∂w, ∂C/∂b) = (−0.2, 5). What does it tell you?

6. A sigmoid neuron has activation a = σ(z) = 0.8. What is the slope ∂a/∂z?

7. In the chain w → z → a → C the local slopes are ∂z/∂w = 3, ∂a/∂z = 0.5 and ∂C/∂a = −2. What is ∂C/∂w?

8. Why are mini-batches used in training?

What you take away

The cost is one number: the average of squared errors over all examples. It is a function of the network's parameters.
The derivative is a slope: how much the output changes when you nudge the input. The sign tells the direction, the size the sensitivity.
Gradient descent repeats the step w ← w − η · slope. The learning rate η must be neither too small nor too large.
The gradient is the list of partial derivatives for all parameters. It points uphill, so we go the opposite way. Its numbers tell which knob matters most.
The chain rule: when a computation is made of links, their slopes multiply.
Mini-batches (SGD) take many cheap, slightly imprecise steps instead of a few expensive ones.

You now hold all the tools. What remains is how to efficiently compute 13,002 partial derivatives at once, when the network is a chain with thousands of branches. The answer is backpropagation: the chain rule applied cleverly from the back to the front, so that nothing is computed twice. That is the topic of the next lecture.

Lecture two is done

Continue to lecture 3

Sources

3Blue1Brown: Gradient descent, how neural networks learn (video, 2017).
3Blue1Brown: Backpropagation calculus (video, 2017).
D. E. Rumelhart, G. E. Hinton, R. J. Williams: Learning representations by back-propagating errors. Nature, 1986.
T. B. Brown et al.: Language Models are Few-Shot Learners (GPT-3), 2020. Batch size and learning rate in Table 2.1.
A.-L. Cauchy: Méthode générale pour la résolution des systèmes d'équations simultanées. Comptes rendus, 1847.