Courses · A course in 7 lectures
How LLMs work
Large language models like ChatGPT look like magic. In this course we take them apart. We start with a single neuron and end with the whole GPT-3 network with 175 billion parameters, which you will be able to click through.
Lectures
First half · How a network learns
1What is a neural network
The neuron, the 1958 perceptron, the limits of one layer, hidden layers and a network that reads digits.
40 minCompleted 2The math of learning: error, slope and gradient
What a network's error is, what a derivative is, and how to walk downhill to a good solution. With a mini exam.
40 minCompleted 3Backpropagation: how a network learns
The algorithm from the Rumelhart, Hinton and Williams paper (1986), step by step on a tiny network.
40 minCompletedSecond half · What is inside an LLM
4Tokens and embeddings: how text becomes numbers
The GPT tokenizer, the space of meanings, positions, encoder and decoder, and the way back from numbers to a word.
40 minCompleted 5Attention and the transformer
Query, key, value, attention heads and the whole block from the paper Attention Is All You Need.
45 minCompleted 6MLP layers: where the model stores facts
The perceptron layers between attention blocks, superposition, and two thirds of all parameters.
35 minCompleted 7All of GPT-3: putting it together
Explore the whole network with 175 billion parameters, how information flows through it and how it is trained.
45 minCompletedWhat the course is built on
Four original research papers and the 3Blue1Brown video series. In the lectures I pick out what matters and translate it into plain language.
F. Rosenblatt. The first model of a neuron that learns from experience. Lecture 1.
D. Rumelhart, G. Hinton, R. Williams. How to train networks with hidden layers. Lecture 3.
A. Vaswani et al. The transformer architecture. Lectures 4 to 6.
T. Brown et al. GPT-3 with 175 billion parameters. Lecture 7.