One word, many meanings
From lecture 4 you know that every token is first turned into a vector: its row of the embedding matrix. In GPT-3 that vector has 12,288 numbers, and directions in that space carry meaning.
But there is a catch. The embedding matrix is just a table. The token mole has one row in it, whatever sentence it appears in:
An animal. Garden, soil, burrow.
A unit in chemistry. 6.02 × 10²³ particles.
A spot on the skin. Doctor, biopsy.
At the start all three “moles” get exactly the same vector. If the model is to predict the next word, that vector has to be nudged towards the right meaning somehow. And the right meaning can only be found in the other words of the sentence.
This is not only about ambiguous words. The word Harry means something different in a sentence about wizards than in one about the British royal family. Eiffel Tower means something different when the word “miniature” stands in front of it. And the last word of a detective novel, “The murderer was …”, has to somehow summarise the whole book if the model is to guess the name.
Key idea
Information has to flow between tokens. The mechanism that does this is called attention. Each token chooses which other tokens to look at, and adds their information into its own vector.
The transformer is a network built around this idea. It was first described in the paper Attention Is All You Need (Vaswani et al., Google, 2017). The T in GPT stands for transformer.
Attention at a glance
Before we compute anything, look at what the result looks like. For every word in a sentence you can say which other words it looks at, and with what weight. The weights of one word always add up to 1, that is 100 %.
A model runs many such “views” at once. Each one is called a head, and each looks for something different in the text. I designed the four below by hand so that they are easy to read. In a trained model they emerge on their own and tend to be much less tidy.
Where the words look
Hover or tap a word. Violet arcs show which earlier words it looks at. Switch between four heads at the top. These are illustrative patterns, not the weights of a real model.
Task: in the “Adjectives” head tap “creature” and “forest”. Then switch to “Pronoun” and tap “it”. Notice that a word only ever looks back, at the words before it. You will find out why in the chapter on masking.
Why does a noun look at adjectives? Because “creature” straight out of the embedding matrix does not know it is fluffy and blue. After attention its vector should point somewhere between “creature”, “fluffiness” and “blueness”. Exactly how that is computed is what the rest of this lecture is about.
Question, label and content: query, key, value
Imagine an archive full of folders. You have a question. Each folder has a label on its spine and content inside. You compare your question with the labels, and from the folders whose label matches you take the content. Attention does the same, only with vectors:
What the token is looking for. For “creature”, say: “Are there adjectives in front of me?”
What the token offers, its label. For “fluffy”: “I am an adjective.”
What the token hands over to whoever looks at it. For “fluffy”: the “fluffiness” direction.
Where do these three vectors come from? They are not stored anywhere. They are computed from the token's embedding by multiplying it with three matrices, exactly like a network layer from lecture 1 (W · a, only without a bias and without an activation function):
How do we tell whether a query “matches” a key? We take their dot product, which you know from lecture 4: multiply the corresponding numbers and add them up. When both vectors point in a similar direction the result is large and positive. When they are unrelated it is close to zero.
The matrices learn themselves
Nobody tells the model “look for adjectives”. The matrices WQ, WK and WV are ordinary weights, set by backpropagation from lecture 3. What individual heads learn to look for is something researchers only find out afterwards, and it is often hard to name.
The whole computation on small numbers
Now let's compute it all. To make it fit on the screen the model is tiny: 4 tokens, embeddings of 4 numbers and one head with dk = 2. GPT-3 does exactly the same, only with 12,288 numbers per token and 128 numbers in each query. I set the matrices by hand so that a noun asks about adjectives.
Attention calculator, step by step
Use Next to walk through the computation. Hover or tap any cell of a result: the numbers it came from light up (violet and teal), and the exact arithmetic is printed below. Cells with a dotted border are editable.
Task: go through all 11 steps with the order “the fluffy blue creature”. In step 7 find the “creature” row: fluffy and blue each get 47 %. Then pick the order “the creature fluffy blue”. The creature now comes before the adjectives, the mask hides them, and it gets nothing from them. Finally, in step 1, change the “fluffiness” of fluffy to 2 and watch the result change.
Let's recap what happened. The “creature” row had a large query q₁ (“I am looking for adjectives”). The adjectives had a large key k₁ (“I am an adjective”). Their dot product was 4, about 2.83 after dividing by √2, and softmax turned that into weights of 0.47 and 0.47. The adjectives' values carry fluffiness and blueness, so the creature's vector gained 0.47 of fluffiness and 0.47 of blueness.
Three things to remember
1. Every query is scored against every key: n tokens give an n × n table. 2. Softmax runs over rows, so the weights of one token sum to 1. 3. The result is added to the original vector. Attention does not replace the vector, it only adjusts it.
Why we divide by the square root of dk
In the calculator we divided the scores by √2. In the original transformer dk = 64, so the division is by 8. In GPT-3 dk = 128 and the division is by about 11.3. Why bother at all?
A dot product is a sum of dk products. The more numbers you add, the larger the results get. The authors say it precisely: if the numbers in the query and the key are random with mean 0 and variance 1, their dot product has variance dk. So the typical size of a score grows like √dk.
And softmax is sensitive to large numbers. A score difference of 1 gives weights in a ratio of about 2.7 : 1. A difference of 10 gives a ratio of 22,026 : 1, practically everything to a single winner. Such a softmax is “saturated”: a small change of the weights changes nothing, the gradient is nearly zero and the model stops learning.
Softmax with and without scaling
One random query and 6 random keys, all numbers random with mean 0 and variance 1. The slider changes the vector length dk. On the left, softmax of the raw scores; on the right, of the scores divided by √dk.
Task: move dk from 2 to 512. On the left the range of the scores keeps growing and one key nearly always swallows almost all the weight. On the right the scores stay around 1 and the weights are shared sensibly. Try other random vectors too.
From the paper (Vaswani et al., 2017)
We suspect that for large values of dk, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients. To counteract this effect, we scale the dot products by 1/√dk.Section 3.2.1, Scaled Dot-Product Attention.
The mask: don't look into the future
You have surely noticed in the demos that words only look backwards. The reason has to do with how GPT is trained.
GPT learns to guess the next token. One text can be turned into many tasks at once: after “The” comes “cat”, after “The cat” comes “sat”, and so on. The transformer computes all positions in parallel in one pass, so a text of 6 tokens gives it 6 training examples at once. With 2,048 tokens that is 2,048 examples from a single text.
But if the position “cat” could look at “sat”, the task would be trivial: the answer would be right in front of it and it would learn nothing. So before softmax, every score where the key comes after the query is overwritten with −∞. Because e−∞ = 0, after softmax these become exactly zero and the remaining weights automatically rescale to sum to 1.
Causal mask
On the left are scores (made-up numbers), on the right the attention weights after softmax. Tap a row and the readout below shows what that token sees and what it has to guess. Toggle the mask on and off.
Task: switch the mask off and look at the “sat” row. The token that has to guess “on” gives a non-zero weight to “on” itself, so it is cheating. Switch the mask on: the upper-right triangle becomes zeros and the other weights in the row grow so the sum stays 1.
Where there is no mask
The encoder of the original transformer (more in chapter 9) has no mask. It processes the sentence to be translated, and the model knows that sentence in full. Looking ahead is fine there.
Many questions at once: multi-head attention
One head asks one question. A text needs many: Who is the subject of this verb? What does “it” refer to? Which adjectives belong to me? What word came right before me? That is why a transformer runs many heads in parallel, each with its own matrices WQ, WK and WV. Each proposes its own change to the vector and all the changes are added up.
Heads side by side
The four heads from chapter two as tables: the row is the word that asks, the column is the word it looks at. A brighter cell is a larger weight. Tap a word or a cell. Below is a counter for the heads of GPT-3.
Task: tap “it” and compare where the same word looks in different heads. Then move the slider to 96 layers and see how many weights attention has in GPT-3 in total.
The original transformer and GPT-3 in numbers
In the 2017 paper each layer has h = 8 heads, the embedding has dmodel = 512 numbers and each head works with dk = dv = 512 / 8 = 64 numbers. GPT-3 has 96 heads in each layer, dmodel = 12,288, and 12,288 / 96 = 128 numbers per head.
The value is a little sneaky. If the value matrix mapped 12,288 numbers straight to 12,288, it would have over 150 million weights. So it is split into two smaller ones: value-down shrinks the vector to 128 numbers and output (value-up) expands it back to 12,288. In the calculator these were the matrices WV and WO.
| GPT-3, one head | Shape | Weights |
|---|---|---|
| Query WQ | 128 × 12,288 | 1,572,864 |
| Key WK | 128 × 12,288 | 1,572,864 |
| Value-down WV | 128 × 12,288 | 1,572,864 |
| Output (value-up) WO | 12,288 × 128 | 1,572,864 |
| One head in total | 6,291,456 | |
| One layer: 96 heads | × 96 | 603,979,776 |
| Whole model: 96 layers | × 96 | 57,982,058,496 |
Almost 58 billion weights, roughly a third of GPT-3's 175 billion parameters. Most of the rest sits in the MLP layers, which we get to in lecture 6.
What the value does to the vector
The output of attention is a change that is added to the embedding. Geometrically that means the vector moves. The value of “fluffy” points in the direction of fluffiness, the value of “blue” in the direction of blueness. The attention weights decide how far the creature's vector moves in each direction.
Moving the embedding
A simplified embedding space with 2 dimensions instead of 12,288. The white arrow is the original vector of the word “creature”. The sliders set the attention weights on “fluffy” and “blue”. The amber arrow is the result.
Task: try to find weights where the amber point lands exactly on “fluffy blue creature”. Then click “As in the calculator”: weights of 0.47 and 0.47 get almost there. Notice that the weights add up to at most 1; the rest goes to the token itself.
This is the essence of attention: it moves information from one token to another. After one layer the vector of “creature” also carries “fluffy” and “blue”. In the next layer a verb can look at it and find out that the creature that found something was fluffy and blue. Layer by layer, the meaning of every token gets refined by its context.
The transformer: attention, MLP and “add & norm”
Attention is one component. The transformer combines it with others into a block, which repeats over and over:
- Attention: tokens exchange information.
- Add & Norm: the result is added to the input and the numbers are normalised.
- Feed forward (MLP): every token is processed on its own, by a plain network from lecture 1. By all accounts this is where facts are stored (lecture 6).
- Add & Norm again.
Add is a residual connection: the output of a sublayer is added to its input, x + Sublayer(x). Thanks to this each layer only adjusts the vector and does not have to build it from scratch. On top of that, during backpropagation the gradient flows straight along this “bypass”, so even very deep networks can be trained.
Norm is layer normalization: the numbers of one vector are shifted and stretched so that they have mean 0 and variance 1. Then they are multiplied by a learned number γ (gain) and shifted by β (bias). Without it, after dozens of layers the numbers could easily blow up to huge values.
Add & Norm on a vector of 5 numbers
x is the sublayer input, Δ is what the sublayer computed. Watch them being added and normalised. The slider makes Δ bigger; γ and β change the result after normalisation.
Task: increase Δ to × 20. The sum x + Δ grows into the tens, but the “normalised” row stays in the same range and still has mean 0 and variance 1. Then change γ and β and see how the mean and variance of the output change.
The full architecture from the paper
The original transformer was built for translation, for example from English to German. That is why it has two parts. The encoder (left) reads the whole input sentence. The decoder (right) writes the translation token by token, looking into the encoder as it goes. GPT took only the right-hand part, without the link to the encoder, and stacked 96 layers of it.
Take the transformer apart
A redrawing of Figure 1 from Attention Is All You Need. Tap any block, the N× label or a caption, and the panel on the right (below on mobile) shows what goes in, what comes out, and how big it is in the original model and in GPT-3.
Task: switch to “GPT: decoder only”. The encoder and the cross-attention fade out and a single column remains, repeated 96×. Then tap Feed Forward and compare its sizes with attention. The original base model had 65 million parameters in total; GPT-3 has 175 billion.
From the paper (Vaswani et al., 2017)
The output of each sub-layer is LayerNorm(x + Sublayer(x)) … To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension dmodel = 512.Section 3.1. GPT-2 and GPT-3 put the normalisation before the sublayer instead: x + Sublayer(LayerNorm(x)).
Cross-attention: the translation looks at the original
The decoder of the original transformer has one more attention layer that GPT does not have. It is called encoder–decoder attention or cross-attention. It is computed just like the one in the calculator, with one difference:
- The queries come from the decoder, that is, from the translation so far.
- The keys and values come from the encoder, that is, from the source sentence.
So every word of the translation can ask: “Which word of the original am I translating right now?” No mask is needed here; the model knows the whole source sentence.
Alignment in translation
Rows are words of the German translation, columns are words of the English sentence. Tap a row to see on the right where that word looks. The numbers are illustrative; real patterns look much like this.
Task: find the German word that looks back into the middle of the English sentence even though it stands at the very end (“gesehen” → “seen”). Word order differs between languages and attention handles it without any rules.
Why attention beat RNNs
Before 2017, text was processed mostly by recurrent networks (RNNs, LSTMs). They read text like a person: word by word, carrying everything they need to remember in a single state vector. This has two big drawbacks:
- It cannot be parallelised. To process the 100th word you must have finished the 99 before it. A graphics card with thousands of cores spends most of its time waiting.
- A long path. Information from word 1 has to pass through 99 steps before it reaches word 100. It gets blurred along the way.
Attention connects every token with every other directly, in one step, and all positions are computed at once. Table 1 of the paper summarises this:
| Layer type | Work per layer | Sequential steps | Longest path |
|---|---|---|---|
| Self-attention | O(n² · d) | O(1) | O(1) |
| Recurrent (RNN) | O(n · d²) | O(n) | O(n) |
| Convolutional | O(k · n · d²) | O(1) | O(logk n) |
n is the number of tokens, d the vector length. The payoff was immediate: the base model trained in 12 hours on 8 P100 graphics cards, and the larger “big” model reached 28.4 BLEU on English-to-German translation after 3.5 days, better than every earlier model.
The price: n²
Attention has a downside too. The score table has n × n cells. GPT-3 has a context of 2,048 tokens, so one head in one layer computes 2,048² = 4,194,304 scores. Double the context and there is four times as much work. That is why models have a limited context length, and why a lot of research goes into making attention cheaper.
2,048 × 2,048 query–key pairs at GPT-3's full context.
96 heads in each of 96 layers. Each computes its own table.
A text twice as long means four times as many scores in every head.
Check what you understood
Six questions. Work two of them out on paper.
1. The query is q = (1, 2) and the key is k = (3, −1). What is the score q · k (without dividing by √dk)?
1 · 3 + 2 · (−1) = 3 − 2 = 1. A dot product: multiply the corresponding numbers and add them up.
2. A row of scores after masking is (1, 1, −∞). What are the attention weights after softmax?
Twice e¹ ≈ 2.72 and once e−∞ = 0. The sum is 5.44, so the weights are 2.72 / 5.44 = 0.5, again 0.5, and 0. The masked token gets exactly zero and the others share the whole weight.
3. Why are the scores divided by √dk?
The dot product of dk random numbers has variance dk. Dividing by √dk brings the scores back to around 1. Softmax makes the weights sum to 1; the mask stops looking ahead.
4. Why does GPT use a mask?
The mask makes it possible to learn at every position of a text at once: each one guesses its own next token and only sees what came before.
5. GPT-3 has 96 layers with 96 heads each. How many heads does it have in total?
96 × 96 = 9,216. Each has 6,291,456 weights, 57,982,058,496 in total, about a third of all parameters.
6. The context grows from 2,048 to 4,096 tokens. By what factor does the number of scores in one head grow?
The score table has n × n cells. 4,096² = 16,777,216, which is four times 4,194,304.
What you take away
- Attention sends information between tokens so that the meaning of each word is refined by its context.
- Query, key, value are three vectors computed from the embedding by multiplying it with the matrices WQ, WK, WV.
- softmax(QKᵀ / √dk) · V: score every query against every key, scale, softmax over rows, weighted sum of the values.
- The mask sets future tokens to −∞ so they get weight 0. This lets GPT learn at all positions at once.
- Multi-head: many heads ask different questions in parallel. GPT-3 has 9,216 heads and about 58 billion attention weights.
- A transformer block = attention + MLP, each with a residual connection and layer normalization. GPT is a stack of 96 such blocks.
Attention handles how tokens talk to each other. But where does a model keep its knowledge, for example that Michael Jordan plays basketball? That is the job of the other half of every block, the MLP layers. They also hold most of the 175 billion parameters. That is the topic of the next lecture.
Lecture five is behind you
Sources
- A. Vaswani et al.: Attention Is All You Need. NeurIPS, 2017.
- 3Blue1Brown: Attention in transformers, step-by-step (video, 2024).
- T. Brown et al.: Language Models are Few-Shot Learners (GPT-3), 2020.