1. What is a neural network – How LLMs work

A task you cannot program

Look at the three threes below. Each is drawn differently, each has different pixels, and yet you instantly see a three. Your brain does it without effort.

Three handwritten threes from the MNIST dataset. Each image is 28 × 28 pixels, that is 784 numbers from 0 (black) to 1 (white).

Now imagine writing a program that receives those 784 numbers and says which digit it is. Try to come up with a rule. “If there is an arc at the top and an arc at the bottom, it is a three”? But what is an arc in the language of pixels? And what if the three is shifted, tilted or thicker? Classic programming with if … else rules breaks down very quickly here.

A neural network takes a different route. We do not write the rules. The network finds them on its own from examples. We only give it a construction kit of very simple parts and thousands of examples with the right answers.

The main idea

A neural network is a function with a lot of adjustable numbers. Numbers go in (pixels, words, sound), numbers come out (the answer). Learning means finding a setting for which the function answers correctly.

In this lecture we look at what such a function looks like. How its numbers get set automatically is the topic of lectures 2 and 3. And the whole course builds toward understanding that ChatGPT is the same idea, just with 175 billion adjustable numbers.

First, the whole picture in two minutes

This short video shows everything we will take apart step by step in this lecture: the neuron, weights, layers and how a signal flows through a network. You do not need to follow every detail yet, just get a feel for it.

An animated summary of the lecture (no sound, with captions). Inspired by the Neural networks series by 3Blue1Brown.

The neuron: a thing that holds one number

The word “neuron” sounds biological, but in artificial networks it is a surprisingly simple thing. A neuron is a place where one number is stored. We call that number its activation. When it is high the neuron is “lit up”, when it is low it is “off”.

Where does the number come from? The neuron computes it from the numbers that flow into it, in three steps:

It multiplies each input by a weight. A weight says how much that input matters. A positive weight supports the neuron, a negative one works against it, and zero ignores the input.
It adds everything up and adds a bias. The bias is a number that sets how easily the neuron lights up even when the inputs are weak.
It passes the result through an activation function. That function decides what goes out. The simplest one is a step: if the sum is above zero the output is 1, otherwise 0.

z = w₁·x₁ + w₂·x₂ + … + w_n·x_n + b → output = f(z) x are the inputs, w the weights, b the bias, f the activation function. This formula is called a weighted sum.

That looks abstract, so let us use a decision you know. Will you go out tonight? It depends on three things, and each carries a different weight for you.

Try it

One neuron decides: go out tonight?

Switch the inputs on the left on and off and move the weights. Watch how the weighted sum changes and when it crosses zero.

Task: set the weights so that you always go out when your friends are going, except when you have an exam tomorrow. Then change only the bias and watch a cautious person turn into a sociable one.

Notice two things. First, the behaviour of the neuron lives entirely in its weights and bias. The same neuron with different numbers decides completely differently. Second, the bias acts as a threshold: the more negative it is, the more “evidence” the neuron needs before it lights up.

The resemblance to the brain

A real neuron receives signals from thousands of other neurons through connections (synapses) of varying strength. When the sum of the signals crosses a threshold, the neuron fires an impulse onward. The artificial neuron is a very rough simplification of that idea. The rest is mathematics, not biology.

The perceptron: a machine that learns from mistakes

A neuron with a step function has a name: the perceptron. In 1958 the psychologist Frank Rosenblatt described it in the paper The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. It is one of the four papers this course stands on.

Rosenblatt asked three questions: how does a living organism sense the world, in what form does it store information, and how does stored information influence its behaviour? At the time many people pictured memory as a filing cabinet: every memory stored somewhere as an image. Rosenblatt argued the opposite.

From the paper (Rosenblatt, 1958)

Information is not stored as a record of individual images. It is contained in the connections, in which pathways between cells have been strengthened.

A free paraphrase of the paper's main thesis. Psychological Review, 65(6), 386–408.

Exactly this holds for neural networks to this day. ChatGPT does not keep the sentences it read stored anywhere. It only has connection weights.

What the original perceptron looked like

Rosenblatt's model had three kinds of units, named after their role:

SSensory units

The “retina”. Points that respond to light. They correspond to our inputs x.

AAssociation units

Cells randomly wired to the retina. They fire when the sum of their inputs exceeds a threshold θ.

RResponse units

The output. They say, for example, “left” or “right”, “square” or “circle”.

The key point was that the strength of the connections between A and R units changed with experience. When the machine answered correctly, the active connections were reinforced. Rosenblatt described this in the language of probability (hence “probabilistic model” in the title) and showed that such a system can learn to tell stimuli apart even when it starts out wired completely at random.

It was not just theory. At the Cornell Aeronautical Laboratory they built the Mark I Perceptron: a machine with a camera of 400 photocells (20 × 20), where the weights were potentiometers turned by electric motors during learning. Newspapers wrote of the embryo of a computer that would walk, talk and be conscious of its own existence. As you will see in a moment, that was rather premature.

The learning rule

In today's notation the perceptron learning rule is very short. We show the perceptron an example and look at what it answered:

Was it right? We change nothing.
It said 0 and should have said 1? The weights were too low. We add a bit of each input to its weight.
It said 1 and should have said 0? The weights were too high. We subtract a bit of each input from its weight.

w ← w + η · (target − output) · x b ← b + η · (target − output) η (eta) is the learning rate: how big a step we take. The term (target − output) is the error: +1, 0 or −1.

A perceptron with two inputs can be drawn. The two inputs are the coordinates of a point in a plane, and the perceptron splits the plane with a straight line: on one side it answers 1, on the other 0. Learning means rotating and shifting that line.

Try it

Teach a perceptron to separate two groups of points

Blue points have target 1, red points 0. Click in the area to add points (pick the colour with the switch). “One step” shows the perceptron one point and adjusts the weights if needed.

Task: run the training on the prepared data and watch the line move with every mistake. Then choose the “XOR” data and see what happens. The line never settles.

If the points can be separated by a line, the perceptron always finds such a line after a finite number of steps. This is a proven theorem (the perceptron convergence theorem). In 1958 that was a huge result: a machine guaranteed to learn a task from examples alone.

What a single layer cannot do

That theorem has a catch in the word “if”. What if the points cannot be separated by a line? Let us look at the simplest tasks there are: logic functions of two inputs.

x₁	x₂	AND (both)	OR (at least one)	XOR (exactly one)
0	0	0	0	0
0	1	0	1	1
1	0	0	1	1
1	1	1	1	0

A single perceptron handles AND and OR easily. XOR (“exactly one of the two”) looks just as innocent. Try setting it by hand.

Try it

Set a perceptron to act as a logic gate

Choose the target function and move the two weights and the bias. The four points are the four rows of the table. The background colour shows what the perceptron answers, the outline of a point shows whether that is correct.

Task: you will hit 4 out of 4 for AND and OR. For XOR you cannot get past 3 out of 4, whatever you do. Try to see why: the points with target 1 lie on a diagonal, and one line cannot separate them from the other two.

This restriction is called linear separability. A single layer can only solve tasks where the answers can be separated by a line (a plane, with more inputs). And most interesting tasks are not like that.

In 1969 Marvin Minsky and Seymour Papert analysed this in the book Perceptrons. They showed mathematically what a single-layer perceptron cannot do. Enthusiasm turned into disappointment, research money dried up, and neural networks nearly vanished for more than a decade. That period is known as the first AI winter.

What “one layer” actually means

A single-layer network has its inputs wired straight to the output neurons. There is nothing in between. Each output neuron is a separate perceptron, so each can draw just one line. More neurons side by side give more lines, but none of them can use the result of another.

The fix: put neurons behind each other

The solution is surprisingly simple. If one neuron can draw one line, take two neurons that draw two lines, and a third that combines their results.

XOR can also be phrased like this: “at least one input is 1, but not both at once”. Those are two simple conditions a perceptron can handle:

The first neuron computes OR: is at least one input 1?
The second neuron computes AND: are both inputs 1?
The output neuron says 1 when the first is on and the second is off.

The neurons between input and output form a hidden layer. It is called hidden because we see its values neither at the input nor at the output, and nobody tells it what to compute.

Try it

A 2-2-1 network solves XOR

The network has two inputs, two hidden neurons and one output. Click a point in the area (or the inputs) and watch which values flow through the network. All 9 numbers can be changed.

Task: click “Show a solution” and go through all four input combinations. Then set the weights to zero and try to build a solution yourself. Notice that each hidden neuron draws one line and the output cuts a band out of them.

This is the most important idea of the whole lecture. The hidden layer translates the input into a new form in which the task is easy. The original points could not be separated by a line. After passing through the hidden layer they can. Deep networks do the same thing many times in a row: each layer prepares the input for the next one.

Why was this not done back in 1958?

The perceptron learning rule only works for one layer. For an output neuron we know what it should have answered. For a hidden neuron we do not know what its “correct” value should have been, so we do not know how to fix its weights. This problem was only solved by the backpropagation algorithm, popularised in 1986 by Rumelhart, Hinton and Williams. All of lecture 3 is devoted to it.

From the step to smooth functions

The step function has one unpleasant property. When you change a weight a little, the output usually does not change at all, and occasionally it jumps straight from 0 to 1. So the network gets no information about whether it is heading in the right direction. For networks to be trainable we need a function that responds smoothly.

Step

0 or 1. The original perceptron. A clear decision, but it cannot be tuned smoothly.

Sigmoid σ

A smooth “S” between 0 and 1. A very negative sum gives almost 0, a very positive one almost 1. Used for decades.

ReLU

Turns negative numbers into 0 and leaves positive ones alone. Simple and fast, the most common today. GPT uses a close relative (GELU).

Try it

What the weight and bias do to a neuron's output

The plot shows the output of a neuron with a single input x. Switch the activation function and change the weight and bias.

Notice: the weight changes the steepness (and flips the plot when negative), the bias shifts the plot left and right. With a sigmoid and a very large weight you get almost a step.

There is one more reason we need an activation function. If neurons only multiplied and added, the whole network, no matter how many layers it had, could be replaced by a single layer. A sum of sums is still just a sum. Only the “bend” in the activation function gives layers their purpose and lets the network draw curves instead of straight lines.

A deep network that reads digits

Back to the task from the beginning. We build a network exactly like the one in the famous 3Blue1Brown video:

784Input layer

One neuron for every pixel of the 28 × 28 image. Its activation is the pixel's brightness from 0 to 1.

16 + 16Two hidden layers

This is where the “thinking” happens. The number 16 was chosen only so the network can be drawn.

10Output layer

One neuron for each digit 0 to 9. The one that lights up the most is the network's answer.

Every neuron is connected to all neurons of the previous layer. Let us count how many adjustable numbers the network has:

weights: 784·16 + 16·16 + 16·10 = 12,960 biases: 16 + 16 + 10 = 42 total: 13,002 13,002 “knobs” that can be turned. The network below has them set by training on 60,000 examples.

The network below is real. I trained its weights on the MNIST data, and on test digits it has never seen it is right about 93% of the time. Everything is computed right in your browser.

Try it

Draw a digit and watch the network work

Draw with your mouse or finger in the black square. On the right you see the activations of all layers. Click any neuron in a hidden or the output layer to see the weights on its inputs.

Try: a seven with and without a crossbar, a digit in the corner, or something that is not a digit at all. The network always answers something. It cannot say “I don't know”, because nobody taught it to.

What do the hidden layers do?

A nice picture is that the network assembles a digit step by step. The first hidden layer would look for short edges, the second would combine edges into loops and strokes, and the output would assemble a digit from loops and strokes. A nine is a loop on top and a stroke below, an eight is two loops.

In the demo above, click the neurons of the first hidden layer and look at their weights. You will mostly not see neat edges. You will see vague blobs. The network found its own solution, which works but is not as tidy as we would hope. That is the honest truth about neural networks: we know exactly how they compute, but often we cannot say “what they think”. In the second half of the course you will see the same holds for large language models.

The whole network is one formula

The computation of one layer can be written very compactly. We put the activations of the previous layer in a column of numbers (a vector a), all the weights in a table (a matrix W) and the biases in another column (a vector b):

a^(new) = σ( W · a^(old) + b ) One row of the matrix W holds the weights of one neuron. Multiplying the matrix by the vector does all the weighted sums at once.

Remember this notation. Exactly this kind of matrix multiplication makes up the vast majority of the computation in GPT, and it is why AI is trained on graphics cards. They are built to multiply large tables of numbers at once.

Where you meet neural networks

The principle is the same everywhere: numbers in, weighted sums, activations, numbers out. What changes is what the numbers mean and how the neurons are wired together.

Images

Unlocking a phone with your face, reading licence plates, finding tumours on X-rays, cars that see pedestrians.

Sound

Speech to text, voice assistants, noise suppression in calls, recognising a song.

Text

Translation, spam filtering, word suggestions on your keyboard and large language models like ChatGPT.

Recommendations

What YouTube, Netflix or an online shop shows you. The network estimates what you will click.

Games and robots

AlphaGo, game-playing agents, walking humanoid robots. Here the network picks the next move or motion.

Science

Predicting the shape of proteins (AlphaFold), the weather, designing new materials and drugs.

The input is always a list of numbers. An image is pixel brightness, sound is air pressure over time, and text? How words become numbers is a separate and very nice topic. We get to it in lecture 4.

Check what you understood

Five short questions. Nothing is sent anywhere, it is just for you.

1. A neuron has inputs x₁ = 1 and x₂ = 0, weights w₁ = 2 and w₂ = 5 and bias b = −3. What is the weighted sum z?

2. Where in a neural network is what it has learned stored?

3. Why can a single perceptron not solve XOR?

4. What would happen if we removed all activation functions from a deep network?

5. A network has 3 inputs, one hidden layer with 4 neurons and 2 outputs. How many parameters (weights and biases) does it have?

What you take away

A neuron holds one number. It computes it as a weighted sum of its inputs plus a bias, passed through an activation function.
The perceptron (Rosenblatt, 1958) is a neuron with a step function that learns from its mistakes. Its knowledge is stored in the weights.
A single layer can only split space with a line, so it cannot even do XOR.
Hidden layers translate the input into a form in which the task is easier. They need a nonlinear activation function for that.
The whole network is a function with parameters. The digit network has 13,002 of them, GPT-3 has 175 billion.

One big question remains open: how do those 13,002 numbers get set? Nobody tunes them by hand. In the next lecture we prepare the mathematics we need for that: what the error of a network is, what a slope and a gradient are, and how to “walk downhill” to a good solution.

Lecture one is behind you

Continue to lecture 2

Sources

F. Rosenblatt: The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review, 1958.
3Blue1Brown: But what is a neural network? (video, 2017).
M. Minsky, S. Papert: Perceptrons. MIT Press, 1969.
Y. LeCun et al.: The MNIST dataset.