Who is to blame?
In the first lecture we built a 2-2-1 network that solves XOR. It has two inputs, two hidden neurons and one output. But we set all 9 of its numbers (6 weights and 3 biases) by hand. In the second lecture you learned how to make a network's error smaller: compute the gradient and take a step against it. One piece is missing. How do you compute that gradient for all the weights, quickly?
For the output neuron it is obvious. We see what it said and we know what it should have said. For a hidden neuron we don't. Nobody told it what value it should have had. When the network answers wrong, is it the output neuron's fault? The first hidden neuron's? The second's? Which of their weights? This is called the credit assignment problem: how to hand out credit and blame.
Has a target. Error = output − target. The perceptron rule from lecture 1 can be used here.
Has no target. We don't know whether it should have fired more or less. The perceptron rule has nothing to compare against.
Computes how much each hidden neuron contributed to the error. The error is sent backwards along the very same weights.
Brute force would work: nudge one weight a little, check whether the error went down, put it back. With 9 weights that's fine. For the digit network with 13,002 parameters, one learning step would need 13,002 runs of the network. For GPT-3 with 175 billion parameters it is unthinkable. Backpropagation gives you all the gradients at once for the price of roughly one pass forward and one pass backward.
Key idea
Backpropagation is not a new way of learning. It is a fast and exact way to compute the gradient. The learning is still gradient descent from lecture 2. Backpropagation just tells it which way to go.
Forward first: a forward pass with real numbers
We will work with one network for the whole lecture. At the start its weights don't know XOR yet. Here they are:
| Neuron | Input weights | Bias |
|---|---|---|
| h₁ (1st hidden) | w₁₁ = 0.5 (from x₁) · w₁₂ = −0.4 (from x₂) | b₁ = 0.1 |
| h₂ (2nd hidden) | w₂₁ = 0.3 (from x₁) · w₂₂ = 0.8 (from x₂) | b₂ = −0.2 |
| y (output) | v₁ = 0.6 (from h₁) · v₂ = −0.5 (from h₂) | b₃ = 0.2 |
The activation is the sigmoid σ(z) = 1 / (1 + e−z), the smooth "S" between 0 and 1 from the first lecture. We feed in x₁ = 1, x₂ = 0. XOR should give 1.
z₂ = 0.3·1 + 0.8·0 + (−0.2) = 0.1 → h₂ = σ(0.1) = 0.5250
z = 0.6·0.6457 + (−0.5)·0.5250 + 0.2 = 0.3249 → y = σ(0.3249) = 0.5805 z is a neuron's weighted sum, h and y are activations. The signal flows left to right, hence "forward pass".
The network said 0.58; it should have said 1. How big is the error? We use the squared difference as in lecture 2, with an extra one half:
Now we want nine numbers: how much the cost C changes when we nudge each of the 9 weights a little. These are the partial derivatives ∂C/∂w from lecture 2. Together they form the gradient.
Intuition: what the output neuron wishes for
Before the formulas, let's do it the 3Blue1Brown way. The output is 0.58 and should be 1. So the output neuron "wishes" its weighted sum z were larger. It has three ways to get there:
- Increase the bias b₃. The most direct. The bias is always added, so it always helps.
- Change the weights v₁ and v₂ in proportion to how brightly their neurons fire. h₁ fires at 0.65, h₂ at 0.52. Raising the weight from the brighter neuron helps more. A weight from a neuron that doesn't fire at all wouldn't help at all. Hence the famous phrase "neurons that fire together wire together": the connection from a neuron that was active exactly when we wanted more gets stronger.
- Change the activations h₁ and h₂ in proportion to the weights. Weight v₁ is positive (+0.6), so it would help if h₁ fired more. Weight v₂ is negative (−0.5), so it would help if h₂ fired less.
The first two we can do directly; they are parameters. The third we can't. The activations h₁ and h₂ are not knobs, they are results of a computation. They can only change through their weights and biases. So the third item becomes a wish for the hidden neurons: "h₁, fire a bit more", "h₂, fire a bit less". And each hidden neuron answers its wish in exactly the same way: the same three options, one layer earlier.
Hence the name
The wish starts at the output and propagates backwards, layer by layer, towards the input. That is literally what back-propagation means. It travels along the very same weights the signal flowed through on the way forward.
That was the wish of one example. The example 1, 1 → 0 will wish for exactly the opposite: a smaller output. When we compute the wishes of all four examples and average them, we get the direction that helps on average. That is exactly the negative gradient from lecture 2.
What δ means
We will write the wish as a number δ (delta) at each neuron: how much the cost changes when we raise the neuron's weighted sum z a little. A negative δ means "raise me and the cost drops". A positive δ means "lower me". Its size says how much it matters.
Backpropagation step by step
This is the core of the whole lecture. Walk through the entire computation yourself. Each step highlights the neurons and weights involved and prints the exact arithmetic. No number is made up; everything is computed live.
One learning step, taken apart into 9 pieces
"Next step" takes you forward, then backward, and finally makes the update. After the last step you can start another round with the new weights. You can change the input and the learning rate at any time.
Task: with input 1, 0, watch steps 4 to 6. Why does δ₂ have the opposite sign to δ₁? (Look at the sign of weight v₂.) Then try input 0, 0 and notice that the gradients of weights w₁₁ to w₂₂ are exactly 0. Finally set the learning rate to 3 and go through a few rounds.
Notice two things. First, the backward pass uses numbers from the forward pass: the activations h₁, h₂, y and the weights. That is why a network remembers all intermediate results during training. Second, the gradient of a weight whose input was 0 is 0. A weight that didn't contribute to the result isn't to blame, so it doesn't change.
The whole algorithm in four equations
What you just clicked through can be written as four equations. They hold for any network, not just ours. The index "in" means the neuron the weight comes from. The expression y(1 − y) is the slope of the sigmoid at y.
The difference from the target times the slope of the sigmoid where the neuron currently sits. Chain rule: ∂C/∂z = ∂C/∂y · ∂y/∂z.
The δ of the neuron the weight leads into, times the activation that arrived over it. For a bias, ain = 1.
Add up the wishes of all neurons h sends its signal to, each weighted by the connecting weight. Then multiply by the slope of the sigmoid.
Gradient descent from lecture 2. η is the learning rate. Done at once for all weights and biases.
For our example 1, 0 → 1 it comes out like this. Check it against the lab above:
∂C/∂v₁ = δ·h₁ = −0.1022·0.6457 = −0.0660
∂C/∂v₂ = δ·h₂ = −0.0536 ∂C/∂b₃ = δ = −0.1022
δ₁ = δ·v₁·h₁(1 − h₁) = −0.1022·0.6·0.2288 = −0.0140
δ₂ = δ·v₂·h₂(1 − h₂) = −0.1022·(−0.5)·0.2494 = +0.0127 Negative δ₁: h₁ should fire more. Positive δ₂: h₂ should fire less, because its weight into the output is negative. Exactly as in the intuition from chapter 3.
Why it works: the chain rule
In lecture 2 you saw that for a chain of functions the slopes multiply. Weight w₁₁ affects the cost through the chain w₁₁ → z₁ → h₁ → z → y → C. Its gradient is the product of the slopes of all links:
= x₁ · h₁(1−h₁) · v₁ · y(1−y) · (y − t) The last two factors are the output's δ, the next two turn it into δ₁, the first is the input. Backpropagation is the chain rule where you compute the shared parts of the product once and reuse them.
This "compute once, reuse" is what makes backpropagation fast. Every weight in the network needs the output's δ. It is computed once and then only passed backwards. If the network had more hidden neurons connected to more outputs, the sum in equation 3 would have more terms, but the principle stays the same.
Don't believe it? Check the gradient yourself
A gradient is a slope. And a slope can be measured: nudge a weight by a small Δ, see how much the cost changed, divide. That is the brute force from chapter one. For a single weight it's fine, and it's a great check that backpropagation computes correctly. People who write neural network code really do this; it's called a gradient check.
Backpropagation vs. measuring by nudging
Tap any of the 9 weights in the network. The chart shows how the cost C changes when you move only that one weight. The slider sets how far you nudge the weight for the measurement.
Task: set Δ to ±1 and see that the measurement is imprecise: the secant drifts away from the tangent. Then make Δ smaller. At Δ = 0.001 the two numbers agree to five decimal places. Also pick weight w₁₂ with input 1, 0: the curve is completely flat and the gradient is 0.
Measuring and backpropagation give the same number. The difference is the cost: measuring needs one extra run of the network per weight, backpropagation gives all of them at once. With 9 weights it doesn't matter. With 175 billion it's the difference between "possible" and "impossible".
Training, live
Now we run the whole thing over and over. The network starts with random weights. In each epoch it goes through all four XOR examples, does backpropagation for each, adds up the gradients and takes one step. That is exactly what the authors of the 1986 paper did. We also add momentum: part of the previous step is added to the new one, like a ball rolling downhill and picking up speed. More about it in chapter 9.
The 2-2-1 network learns XOR on its own
Line thickness is the size of a weight, color is its sign (blue positive, red negative). Below you see what the network answers across the whole square of inputs, and how the cost goes down.
Task: run the training and watch the green band form around the points 0, 1 and 1, 0. Notice the plateau in the cost curve: for a long time nothing happens, then the cost suddenly drops. Then turn momentum off (α = 0) and compare the number of epochs. Finally click "Show a stuck case".
When the network gets stuck
Some random starts end badly. The cost drops to 0.25 or 0.33 and stops moving. The network might, for example, answer almost correctly on three points and say 0.5 on the fourth, or both hidden neurons end up saturated, where the slope of the sigmoid is nearly zero. The gradient there is tiny and gradient descent has nowhere to go. Such a place is called a local minimum or a flat plateau.
With our small network this happens for roughly one random start in ten. The fix is simple: new random weights. The authors of the paper honestly wrote that local minima are the method's biggest weakness, but that in practice the network rarely ends up in a poor one. For large networks it's even less of a problem: in a space with billions of dimensions, almost every "valley" has a way out somewhere.
A plateau is not being stuck
If the cost curve sits at 0.5 for a long time and then drops, that wasn't a local minimum, just a flat stretch. The hidden neurons were "looking for a job". Once they found one, the cost fell fast. A stuck network stays on the flat stretch even after thousands of epochs.
What did the hidden neurons learn?
In lecture 1 we showed that a hidden layer translates the input into a new form in which the task is easy. There we set it by hand (OR and AND). Now let's look at what the hidden neurons found on their own, trained only by backpropagation.
This is the answer to the question from chapter one. Nobody told the hidden neurons what to compute. Backpropagation sent them wishes, and they gradually shifted so that their output was useful to the next layer. The hidden neurons build their own representation of the input. That was exactly the main message of the 1986 paper.
The paper that started it all (1986)
In October 1986 the journal Nature published a four-page paper, Learning representations by back-propagating errors. The authors: David Rumelhart, Geoffrey Hinton and Ronald Williams. It is the second of the four papers this course is built on.
From the paper (Rumelhart, Hinton, Williams, 1986)
As a result of the weight adjustments, internal hidden units which are not part of the input or output come to represent important features of the task domain, and the regularities in the task are captured by the interactions of these units.Close paraphrase of the abstract. Nature 323, 533–536 (1986).
The word representations in the title is no accident. The authors stressed that older methods, such as the perceptron rule, could only learn the weights leading directly to the output. Hidden units had to be set by hand. The new method showed for the first time that a network creates useful internal "concepts" on its own.
The equations from the paper
The paper uses slightly different symbols from ours. xj is the weighted sum of unit j, yj its output, wji the weight from unit i to unit j, dj the desired output. The paper handles biases with a trick: each unit gets one extra input that always has the value 1.
∂E/∂xj = ∂E/∂yj · yj(1 − yj)
∂E/∂wji = ∂E/∂xj · yi
∂E/∂yi = Σj ∂E/∂xj · wji The backward pass. ∂E/∂xj is our δ. The second line is equation 1, the third is equation 2, and the fourth together with the second gives equation 3 from chapter 5.
Why momentum
The authors first described a plain step against the gradient and immediately added that it is slow. Their improvement is the second formula above: α times the previous step is added to the new step. In a direction where the gradient keeps pointing the same way, steps add up and the network speeds up. In a direction where the gradient flips back and forth, the steps cancel out. It's easiest to see in a narrow valley.
Momentum in a narrow valley
The ellipses are contour lines of the cost E(w₁, w₂) = ½(0.5·w₁² + 10·w₂²). The valley is 20 times steeper top to bottom than left to right. The blue path is plain gradient descent, the amber one has momentum. Tap the plot to change the start.
Task: at α = 0.9 compare the number of steps to the goal. Then raise the learning rate to 0.21: the blue path starts zig-zagging across the valley and eventually flies off, because the step is too big for the steep walls. The amber one holds. Also try α = 0.95 and watch it start swinging around the minimum.
Two experiments from the paper
The input is 6 bits. The network must say whether the string is symmetric about its middle, like 101101. Two hidden units were enough. It learned this after 1,425 sweeps through all 64 possible inputs with ε = 0.1 and α = 0.9.
Two families with the same structure, one English and one Italian, 24 people in total and 12 relationships (father, aunt, nephew…). The network got triples like "Colin has-aunt ?" and had to fill in the person.
For symmetry the network found an elegant solution. Weights at mirror positions have equal size but opposite sign, so a symmetric input cancels out and both hidden units stay off. The weight sizes on one half are in the ratio 1 : 2 : 4, so no non-symmetric input can cancel out. Nobody hinted at this.
In the family trees each input person was coded by one neuron out of 24, with no information about who is who. Still, the 6 hidden units that encoded the input person developed meaningful features: one distinguished whether the person was English or Italian, another encoded the generation (grandparents, parents, children) and another the branch of the family. Just like our h₁ and h₂ for XOR, only with a more understandable meaning.
The authors were honest about the weaknesses too. They wrote that the method can get stuck in a local minimum, but that in practice this rarely happens. And they wrote that their procedure in its current form is not a plausible model of learning in brains. It's an engineering tool, not biology.
Who invented backpropagation?
The honest answer: not the authors of this paper, at least not first. Computing derivatives backwards through a chain of operations was described by the Finnish student Seppo Linnainmaa as early as 1970 (today it's called reverse-mode automatic differentiation). Paul Werbos proposed applying it to neural networks in his 1974 PhD thesis. The 1986 paper's contribution was to show clearly that hidden layers learn useful representations this way, to demonstrate it convincingly in experiments, and to make the method popular. Since then neural networks have been trained almost exclusively like this.
Geoffrey Hinton
The paper's second author kept working on neural networks even when it was unfashionable. He later came to be called the "godfather of AI". In 2018 he received the Turing Award together with Yoshua Bengio and Yann LeCun, and in 2024 the Nobel Prize in Physics together with John Hopfield.
From 9 parameters to 175 billion
Everything we did with the 2-2-1 network holds unchanged for big networks. The digit network from lecture 1 has 13,002 parameters. It has 10 output neurons, so the output δ is 10 numbers. Each neuron in the second hidden layer is connected to all 10 outputs, so in equation 3 it adds up 10 wishes. And so on, all the way back to the input.
What backpropagation wishes for in the digit network
The real 784-16-16-10 network from lecture 1. Pick a digit. You'll see how brightly each output neuron fires and what change backpropagation asks of it (−δ). Below that, one step further back, are the 16 neurons of the second hidden layer.
Notice: the correct digit (highlighted) almost always wants to go up and the others down. When the network is confident and right, the wishes are small. The biggest wishes come from neurons that fire wrongly but are not saturated. Try more samples and find one the network gets wrong.
Mini-batches: learning in small bites
In our lab, each epoch added up the gradients of all 4 examples. MNIST has 60,000 examples, and the text used to train GPT-3 had hundreds of billions of tokens. Going through everything before each step would take forever. So we use stochastic gradient descent from lecture 2: randomly pick a small batch (a mini-batch), say 32 or 512 examples, run backpropagation on each, average the gradients and take a step. The direction isn't perfect, but it's good enough, and you get to take many more steps.
4 examples. A few hundred epochs. Your browser does it in a fraction of a second.
60,000 examples, mini-batches. A few minutes on an ordinary computer.
About 300 billion tokens. Thousands of graphics cards for weeks. Still the same algorithm.
How much does backpropagation cost? The backward pass does the same kind of work as the forward pass: one multiply and add per weight, just in the opposite direction. The authors of the paper stressed this: the backward pass has the same order of complexity as the forward pass. In practice the backward pass costs roughly one to two forward passes. So the gradient of all 175 billion parameters costs about as much as two or three runs of the network, not 175 billion runs. Without this property no large language model would exist.
One algorithm for everything
GPT doesn't use the sigmoid but other activation functions. It doesn't use squared error but a different cost (cross-entropy). It has attention and millions of neurons. Backpropagation doesn't care. As long as you can state the slope of every link in the chain, the chain rule assembles the gradient of the whole. Modern libraries (PyTorch, JAX) do it automatically.
Check what you've understood
Six questions, some with a small calculation. Pen and paper help.
1. A sigmoid output neuron gave y = 0.8, the target is t = 1. What is its δ = (y − t)·y(1 − y)?
(0.8 − 1)·0.8·0.2 = −0.2·0.16 = −0.032. A negative δ means: raise the weighted sum and the cost drops.
2. A hidden neuron has activation h = 0.5. The output neuron has δ = 0.1. What is the gradient of the weight v connecting them?
∂C/∂v = δ·h = 0.1·0.5 = 0.05. A weight's gradient is the δ of the neuron it leads into, times the activation that arrived over it.
3. The output has δ = 0.2, the weight from a hidden neuron to the output is v = −1, and the hidden neuron has h = 0.5. What is the hidden neuron's δ?
δh = δ·v·h(1 − h) = 0.2·(−1)·0.25 = −0.05. The output wants to go down, but the weight is negative, so the hidden neuron should fire more.
4. Why can't the perceptron rule train hidden neurons?
The perceptron rule needs the error (target − output) right at the neuron. A hidden neuron has no target. Backpropagation computes a δ for it from the output's error instead.
5. The network's input was x₁ = 0. What is the gradient of every weight leaving x₁?
∂C/∂w = δ·x₁ = δ·0 = 0. A weight that didn't contribute to the result isn't to blame. You saw it in the lab with input 0, 0.
6. What was the main message of the 1986 paper by Rumelhart, Hinton and Williams?
The title is Learning representations. Hidden units found, for example, nationality and generation in the family tree on their own. The authors also explicitly wrote that it is not a plausible model of the brain.
The first half of the course is behind you
This was the last lecture about how neural networks work in general. Look at everything you now know:
- Lecture 1: a neural network is a function with many adjustable numbers. A neuron takes a weighted sum, adds a bias and passes it through an activation function. Hidden layers translate the input into a form in which the task is easy.
- Lecture 2: learning means shrinking the error. The gradient points where the error grows fastest, and gradient descent goes the other way. The learning rate sets the step size; mini-batches make it faster.
- Backpropagation computes the gradient of all weights at once. The error spreads as δ from the output backwards along the same weights. A weight's gradient = δ of the neuron it leads into × the activation on its input.
- The chain rule is the whole secret. Backpropagation applies it so that each intermediate result is computed only once, which is why it costs only about one or two runs of the network.
- Hidden neurons find representations on their own that make the task simpler. That was the main discovery of the 1986 paper, and it is why deep networks work.
- The same algorithm trains a network with 9 parameters and GPT-3 with 175 billion. The size changes, not the principle.
In the second half of the course we move to language. First we'll find out how text turns into numbers a network can compute with: tokens and embeddings. Then comes attention, the heart of the transformer, and the MLP layers, where most of a model's knowledge is stored. At the end we'll count all 175 billion parameters of GPT-3. And remember: every one of them was set exactly the way you saw today on nine numbers.
Lecture three is done
Sources
- D. E. Rumelhart, G. E. Hinton, R. J. Williams: Learning representations by back-propagating errors. Nature 323, 533–536, 1986.
- 3Blue1Brown: What is backpropagation really doing? (video, 2017).
- 3Blue1Brown: Backpropagation calculus (video, 2017).
- S. Linnainmaa: The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's thesis, University of Helsinki, 1970.
- P. J. Werbos: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard, 1974.
- Y. LeCun et al.: The MNIST dataset.