7. All of GPT-3: putting it together – How LLMs work

GPT-3 in one sentence

At the start of the course we said that a neural network is a function with lots of adjustable numbers. Now you know exactly what that means for GPT-3:

Key idea

GPT-3 is a function with 175,181,291,520 knobs. It takes at most 2,048 tokens and returns a probability distribution over 50,257 possible next tokens. Nothing more. Everything else, essays and translations included, comes from calling this function again and again.

P(next token | up to 2,048 previous tokens) = GPT-3_θ( tokens ) θ (theta) stands for all 175 billion parameters at once. The output is 50,257 numbers between 0 and 1 that sum to 1.

You have already seen every part of this machine. Here is a quick map of the course. Tap a tile if you want a refresher.

1Neurons and layers

Weighted sum, bias, activation. A digit network with 13,002 parameters.

2The math of learning

Error, gradient, walking downhill, mini-batches.

3Backpropagation

The chain rule computes the gradient for every weight at once.

4Tokens and embeddings

BPE, a vocabulary of 50,257, vectors of 12,288 numbers, softmax and temperature.

5Attention

Q, K, V, the mask, 96 heads, residual connections and LayerNorm.

6MLP layers

12,288 → 49,152 → 12,288, facts, superposition, the residual stream.

In this lecture we wire it all together. First you take one pass through the whole model, then we count where the 175 billion parameters sit and show how those numbers got set. At the end you open up a real small GPT that I trained the same way.

One pass through the model, station by station

Below is the whole GPT-3 architecture as a map. It is exactly the machine from the paper Language Models are Few-Shot Learners (Brown et al., 2020): 96 blocks, d_model = 12,288, 96 heads of 128 dimensions, a context of 2,048 tokens. Information flows from top to bottom and then loops back to the start.

Try it

GPT-3 explorer

Tap any component and the panel next to the map (below it on a phone) shows what it does, the shapes of its input and output, and how many parameters it has. Blocks, attention and the MLP can be opened. The “Follow one token” mode takes a sentence through the whole model step by step.

Task: start “Follow one token” and go through all 17 steps. Then open a block on the map, open attention inside it and pick any head. Notice that one head has only 6,291,456 parameters, but there are 96 × 96 = 9,216 heads.

A few things from the map worth remembering:

The shape never changes. An n × 12,288 table enters a block and the same shape leaves it. That is why 96 blocks can be stacked.
The residual stream is the backbone. Attention and the MLP overwrite nothing, they only add. Each block reads what the earlier ones left there and adds its own correction.
Attention mixes information between tokens, the MLP processes each token on its own. These two operations alternate 96 times.
The output depends only on the last vector. When generating, the unembedding only needs the row of the last token. The other positions are still computed, because attention draws on them.

From the paper (Brown et al., 2020, section 2.1)

We use the same model and architecture as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer.

Quoted from section 2.1, citation omitted. “Pre-normalization” means LayerNorm comes before attention and before the MLP, not after them. That is exactly what you see in the block map.

Where the 175 billion parameters live

In lecture 1 we counted the parameters of the digit network by hand: 13,002. GPT-3 works the same way, we just multiply bigger numbers. The table below follows the tally from the 3Blue1Brown videos. It splits the parameters into eight groups of matrices and ignores biases, LayerNorm and positional embeddings, which together make up less than 0.1%.

Group	Formula	Parameters	Share
Embedding W_E	12,288 × 50,257	617,558,016	0.35%
Key W_K	128 × 12,288 × 96 heads × 96 layers	14,495,514,624	8.27%
Query W_Q	128 × 12,288 × 96 × 96	14,495,514,624	8.27%
Value W_V	128 × 12,288 × 96 × 96	14,495,514,624	8.27%
Output W_O	12,288 × 128 × 96 × 96	14,495,514,624	8.27%
MLP up-projection	49,152 × 12,288 × 96	57,982,058,496	33.10%
MLP down-projection	12,288 × 49,152 × 96	57,982,058,496	33.10%
Unembedding W_U	50,257 × 12,288	617,558,016	0.35%
Total		175,181,291,520	100%

Three things jump out of the table. Two thirds of the model is MLP. One third is attention. And the embedding plus unembedding, the whole “vocabulary”, is less than one percent. GPT-3 is not big because it knows many words, but because it has a great many very wide layers.

The explorer above also has a “Where the 175 billion are” mode that shows the same numbers as areas. Now build your own model. The same formulas work for any GPT.

Try it

Build your own GPT

Move the number of blocks, the width d_model, the number and size of heads, the vocabulary and the context. The bars show the eight matrix groups, and the total uses the same formula as the table. The presets are all eight models from Table 2.1 of the GPT-3 paper.

Task: click Small, XL and 175B in turn and compare the total with the paper's value. Then, on 175B, double d_model. The parameters in the blocks quadruple, because every matrix grows in both dimensions. Doubling the number of blocks only doubles them.

Why the total doesn't match the paper exactly

The paper gives rounded numbers (125M, 175.0B) and doesn't spell out exactly what it counted. GPT-2 uses the same matrix for the embedding and the unembedding, so it is counted only once. That is why the demo also shows a “detailed count” in GPT-2 style: one shared vocabulary matrix, plus positions, biases and LayerNorm. For small models it matches the paper much better; for 175B the two ways of counting differ by less than half a percent.

How GPT-3 learned its 175 billion numbers

At the start of training every parameter is a small random number. The model writes nonsense. After that, nobody shows it “correct answers” to questions. The task is much simpler and can be made from any text: guess the next token.

Every position in a text is one example

Take the sentence “The capital of France is Paris”. It gives you five tasks at once: after “The” comes “ capital”, after “The capital” comes “ of”, and so on. Thanks to the mask in attention (lecture 5) a token cannot look ahead, so the model can solve all positions at once in a single pass without cheating anywhere. One sequence of 2,048 tokens is therefore 2,048 training examples. This is called self-supervised learning: the correct answers are right there in the text.

The loss: −ln p of the correct token

For each position the model returns probabilities. We care about just one of them: how much it gave to the token that really came next in the text. If it gave it 1, the loss is 0. If it gave it almost zero, the loss shoots towards infinity. This function is called cross-entropy.

loss = −ln p(correct token) averaged over all positions in the batch ln is the natural logarithm. At p = 1 the loss is 0, at p = 0.5 it is 0.69, at p = 0.01 it is 4.6.

Try it

The loss of one prediction

The model is continuing “The capital of France is”. Pick which token really came next in the training text, and move the probability the model gave it. The probabilities are illustrative.

Task: pick “␣Lyon”. The model gave it only 0.02, so the loss is 3.9. Then move p towards 1 and watch the loss drop to zero. Training does nothing else: it changes the weights so that, on average, you slide down this red curve.

One learning step

A learning step is exactly the same cycle as in lectures 2 and 3, just with enormous numbers. A batch of text, a forward pass, the loss, backpropagation, a small update of all weights. GPT-3 processed 3.2 million tokens per batch and saw 300 billion tokens in total. That is roughly 300,000,000,000 / 3,200,000 ≈ 93,750 steps. It is only an approximate number, because the batch size was ramped up gradually at the start of training.

Try it

The training loop

The dot travels around the five stations of one step. Switch between the real numbers of GPT-3 and the mini-GPT from this lecture. The “500 steps at once” button speeds up the counter.

Notice: in step 4 the gradient is computed for all 175 billion parameters at once, not one at a time. Without the backpropagation from lecture 3, a single step would be unthinkably expensive.

Details from appendix B of the paper: the Adam optimizer (β₁ = 0.9, β₂ = 0.95), a learning rate of 0.6 × 10⁻⁴ with a linear warmup at the start and cosine decay down to 10%, gradient clipping at norm 1.0, and weight decay of 0.1. Adam is an improved gradient descent: for every parameter it keeps running averages of the gradient and uses them to choose the step size. At heart it is still the idea from lecture 2: step against the gradient.

What it cost

According to the paper, training GPT-3 175B took about 3.14 × 10²³ floating-point operations. You can estimate it by hand too: forward plus backward cost roughly 6 operations per parameter per token, so 6 × 175 billion × 300 billion ≈ 3.15 × 10²³.

What learning looks like

We can't see the learning curve of GPT-3, but we can see the one of the mini-GPT from this lecture. I trained it the same way: next token, cross-entropy, backpropagation, Adam with warmup, cosine decay and weight decay 0.1. Here the tokens are single characters, and the text is eight Shakespeare plays. During training I recorded the loss and every now and then let the model write something.

Real data

The mini-GPT learning curve

Red is the loss on the training batches, the teal dots are the loss on text the model never saw during training. Use the slider or click the chart to pick a step, and read below the chart what the model was writing at that moment.

Task: go through the steps from 0 to the end. At first it is a random mix of characters. Then spaces and vowels appear, then short words, character names in capitals, and finally lines that look like a play. Grammar and meaning are still weak. The model is a million times smaller than GPT-3.

What GPT-3 read

A model is as good as the data it predicted the next token on. The authors took the Common Crawl internet archive, filtered out low-quality pages, removed duplicates and added higher-quality collections: WebText2, two book corpora and the English Wikipedia.

Try it

The training data mix

Switch between how big each dataset is, how much weight it had in training, and how many times the model read it. The numbers are from Table 2.2 of the paper.

Notice: Wikipedia has only 3 billion tokens, but the model saw it 3.4 times. Common Crawl is 136 times bigger, yet the model did not see even half of it. The authors deliberately favoured quality over quantity.

300Btokens during training

With the common estimate of 0.75 words per token, that is about 225 billion words.

≈ 1,700years of reading

That is how long reading it would take at 250 words per minute, with no sleep and no breaks.

570 GBCommon Crawl after filtering

Out of the original 45 TB of compressed text from 2016 to 2019.

Generation: one token, then again

A trained model can do exactly one thing: given a text, return probabilities for the next token. How does a whole answer come out of that? Simply with a loop, the one you saw in the explorer as the yellow arrow:

Run the whole text so far through the model and take the probabilities from the last position.
Draw one token from them. Temperature (lecture 4) decides whether it sticks to the most likely ones or takes risks.
Append the token to the end of the text and go to step 1. Stop at a special end token or at a length limit.

Three important consequences follow:

Answers differ. Unless the temperature is zero, the same question gets a different draw each time, and so a different answer.
The model doesn't plan ahead in the text. Each token is produced separately. Any “plan” has to be encoded in the vectors of the residual stream.
What falls out of the context doesn't exist. GPT-3 sees at most 2,048 tokens at once. Older tokens are simply invisible to it.

A chat is just a document

When you talk to a chatbot, the model doesn't see a “conversation”. It sees one long text: some opening instructions, your messages and its own previous replies, separated by agreed-upon markers. And it continues that text, token by token. Chatbots like ChatGPT are additionally fine-tuned after the base training, on examples of good answers and on ratings from people, so that they continue such documents helpfully. The underlying machine is the same.

A real mini-GPT you can open up

You can't download GPT-3 into a browser: its weights alone would take hundreds of gigabytes. So I trained a miniature with the same architecture: a token embedding and a learned positional embedding, blocks with LayerNorm before attention and before the MLP, causal multi-head attention, an MLP 4 times wider with GELU, a final LayerNorm, unembedding and softmax. The only difference is size:

65tokens in the vocabulary

A token is one character: letters, space, newline and punctuation.

3 × 4blocks × heads

d_model = 64, each head has 16 dimensions, the MLP has 256 neurons.

64characters of context

The model sees only the last 64 characters at once.

162,496parameters

Including biases, LayerNorm and positions. By the 3Blue1Brown tally: 155,776.

I wrote the training in plain JavaScript, with no libraries. The forward and backward passes are hand-written and I checked the gradients numerically, just like in lecture 3. The model saw 4,000 steps of 32 snippets of 64 characters, about 8.2 million characters, and the loss on text it never saw during training fell from 4.18 to 1.53. The whole training took about 27 minutes on an ordinary 6-core CPU. The result writes text that looks like Shakespeare until you start reading it: the words mostly exist, the character names fit, the meaning doesn't.

Try it

Open up the mini-GPT

Type some text and follow every step of the computation with real numbers: tokens and their IDs, the embedding, the residual stream after each block, the attention patterns of all 12 heads, the activations of the 256 MLP neurons and the probabilities of the next character. The “Generate” button adds characters one by one and every picture updates as it goes.

Task: type “ROMEO.” and a newline and let the model generate. Then tap a character in the middle of a word and look at the attention heads in block 1. Some look only at the previous character, others at the start of the word or at the newline. Finally raise the temperature to 2 and generate again: the text falls apart into random letters.

This is the whole principle of GPT-3 in miniature. Grow d_model from 64 to 12,288, the blocks from 3 to 96, the heads from 4 to 96, swap characters for 50,257 BPE tokens and 8 million characters for 300 billion tokens, and you get GPT-3. Nothing else changes. The forward-pass code would stay almost the same.

Learning without changing weights: zero-, one- and few-shot

The main discovery of the GPT-3 paper is in its title: Language Models are Few-Shot Learners. The older approach was this: for every task (translation, questions, classification) you train the model further on thousands of examples. That is called fine-tuning and it changes the weights. GPT-3 showed that a large model only needs the task to be described, with a few examples shown right in the text.

Translate English to French: sea otter => loutre de mer peppermint => menthe poivrée plush giraffe => girafe peluche cheese => fromage

This prompt is similar to Figure 2.1 in the paper. The model doesn't know it is “doing translation”. It sees a document in which the pattern “English word => French word” repeats, and the most likely continuation is to keep it going. The authors call this in-context learning: learning from the context.

Try it

Zero-shot, one-shot, few-shot

Switch how many examples the model gets in the prompt. Watch what changes in the model.

Notice: the number of changed weights is always 0. The only difference is the input text. The paper shows that accuracy grows with the number of examples and with model size, and that the largest model benefits the most.

From the paper (Brown et al., 2020)

For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model.

From the abstract. For few-shot the authors usually gave 10 to 100 examples, as many as fit into 2,048 tokens.

How can this work? We don't know exactly. The authors themselves write that it is unclear whether the model really “learns” the task from the examples or merely recognizes a task it saw in the training data. What is certain is that all the work happens in attention and the MLPs during a single forward pass.

Scale: from 13,002 to 175 billion

All the networks in this course are built from the same parts: matrix-vector multiplication, a nonlinearity, the gradient, backpropagation. They differ mainly in size. And the sizes are so different that you can't see them on an ordinary axis. Hence a logarithmic axis, where each tick means 10× more.

Try it

From the digit network to GPT-3

At the top are the parameter counts of five networks. Below, move the context length and watch how many attention scores must be computed.

Task: set the context to 4,096 tokens, twice as much as GPT-3. The number of attention scores grows almost four times. This is why long contexts are expensive.

Attention compares every token with every earlier one, so the work grows with the square of the context length, n². That is also why GPT-3 uses locally banded sparse patterns in half of its layers: there a token only looks at nearby tokens.

What came after GPT-3? Newer models such as GPT-4 build on the same idea of a transformer that predicts the next token. They are bigger, trained on more and better data, and after the base training they are further tuned to give helpful and safe answers. For GPT-4, OpenAI did not publish the size or the exact architecture, so we won't guess.

What we know and what we don't

At the end of the course it's fair to say where the limits of our understanding are.

We know exactly what the model computes. Every multiplication and every number can be printed out, as in the mini-GPT above. There is no magic.
We don't know well why it works. From 175 billion numbers we can't easily read where the fact about Paris is stored or how the model adds numbers. The field of interpretability studies this: it looks for what individual neurons and directions in the residual stream mean. Superposition from lecture 6 is one of the reasons it is hard.
Hallucinations are not a malfunction but a side effect. The model always returns some probabilities, and a next token can always be drawn. It has no special “I don't know” output, just as the digit network from lecture 1 couldn't say that something isn't a digit. Fluent, confident text is not necessarily true.
The model knows nothing after its data ends. It only knows texts from before training. What happened later, it can only guess.

The GPT-3 paper admits this itself: its samples sometimes repeat themselves, lose coherence over longer passages and contradict themselves. The authors also devote a whole chapter to risks, such as biases taken over from the data.

Final quiz for the whole course

Eight questions spanning all seven lectures. Nothing is sent anywhere.

1. What exactly does GPT-3 return after one forward pass?

2. Where are most of GPT-3's parameters?

3. The model gave the correct next token a probability of 0.25. What is the loss (cross-entropy)?

4. Why can every position in a text be a separate training example, all in a single pass?

5. What changes in GPT-3 when you give it five examples of a task in the prompt (few-shot)?

6. What does backpropagation compute in one training step of GPT-3?

7. What do the digit network from lecture 1 and GPT-3 have in common?

8. What happens when you raise the temperature during generation?

What you take away from the whole course

GPT-3 is a function with 175,181,291,520 parameters (by the 3Blue1Brown tally). From at most 2,048 tokens it produces probabilities for 50,257 possible next tokens.
The flow of information: tokenizer → embedding + position → 96 blocks (LayerNorm, attention, add, LayerNorm, MLP, add) → LayerNorm → unembedding → softmax → token → again.
Two thirds of the parameters are in the MLPs, a third in attention and less than a percent in the vocabulary.
Training is next-token prediction on 300 billion tokens: cross-entropy, backpropagation through 96 blocks, and Adam. Roughly 93,750 steps of 3.2 million tokens.
Generation is a loop. A chat is just a document the model continues.
In-context learning: examples in the prompt change the behaviour without changing a single weight.
The mini-GPT with 162,496 parameters is the same machine, just about a million times smaller. And it writes something that looks like Shakespeare.

You have completed the course

You've gone all the way from a single neuron to a model with 175 billion parameters. Next time you see a chatbot write an answer word by word, you'll know exactly what is going on: 96 blocks, attention, MLPs, a softmax and a draw. Again and again.

Back to the course overview

Sources

T. Brown et al.: Language Models are Few-Shot Learners. NeurIPS 2020 (the GPT-3 paper; Tables 2.1 and 2.2, appendix B).
A. Vaswani et al.: Attention Is All You Need. NeurIPS 2017 (Transformer base: 6 layers, d_model = 512, 65 million parameters).
3Blue1Brown: Transformers, the tech behind LLMs (video, 2024).
3Blue1Brown: Attention in transformers, step-by-step (video, 2024).
3Blue1Brown: How might LLMs store facts (video, 2024). The source of the 175,181,291,520-parameter tally.
W. Shakespeare: The Complete Works, Project Gutenberg #100 (training text for the mini-GPT, public domain).