4. Tokens and embeddings: how text becomes numbers – How LLMs work

From digits to language

In the first three lectures the network took 784 pixels and returned 10 numbers: how much the image looks like a zero, a one, a two… It learned this on its own, with gradient descent and backpropagation. A language model works surprisingly similarly. Only what goes in and what comes out changes.

GPT stands for Generative Pretrained Transformer:

Generative

It generates new text. It does not pick from ready-made answers, it builds them piece by piece.

Pretrained

It was first trained on a huge amount of text. GPT-3 saw 300 billion tokens during training.

Transformer

A type of neural network from 2017. We open up its heart in lectures 5 and 6.

Key idea

A language model is a function: text so far → probability of every possible next token. Nothing more. All the “writing” is a loop around this one function.

The loop has four steps:

The model gets the text so far. Your question, for example.
It returns probabilities. For GPT-3 that is 50,257 numbers, one for each token it knows.
One token is sampled according to these probabilities. Not always the most likely one.
The token is appended to the text and everything starts again from step 1.

Try the loop on a toy. Instead of 175 billion parameters it only has a table that counts which word followed the last two words in a short text. The loop, however, is exactly the same.

Try it

The generation loop: predict, sample, append

On the right you see the probabilities of the next token. Sample one at a time, let it generate automatically, or click a bar to pick a token yourself. Blue tokens are the context the toy predicts from.

Task: set the temperature to 0.2 and generate. The text starts repeating because the same option keeps winning. Then set it to 2 and watch the probabilities even out and the text get more chaotic. The toy only sees two words back, so it quickly loses the thread. GPT-3 sees 2,048 tokens.

The toy has one fundamental flaw: it only knows word pairs it has already seen. A real model has to cope with text it has never seen before. That is why it cannot be a table. It is a network that first turns text into numbers and then computes with them. The whole path looks like this:

Text“The cat lay down on the”
Tokenspieces of wordschapters 2 and 3
Vectorsembedding + positionchapters 4 to 7
96 layerstokens “talk to each other”lectures 5 and 6
Logitsa score for each of 50,257 tokenschapter 9
Softmaxprobabilities → samplingchapter 9

The path of text through GPT-3. This lecture covers the input and the output. The middle, where the “thinking” happens, stays a black box for now.

Why not letters or whole words

The network needs a fixed list of pieces that every text is made of. We call this list the vocabulary and its items tokens. How big should the pieces be? There are three options:

Letters

The vocabulary is small, about a hundred characters. But the text gets very long and a single letter means almost nothing on its own.

Whole words

The text is short, but the vocabulary is enormous. Every word has many forms: run, runs, running, ran… And what about a typo or a new name?

✓Pieces of words

A compromise. Common words are a single token, rare ones are assembled from smaller pieces. Nothing is “unknown”.

GPT uses the third option. The pieces are found by an algorithm called BPE (Byte Pair Encoding). It started as a data compression method in 1994. In 2016 Sennrich and colleagues used it for machine translation, and since then nearly every language model has used it. It is surprisingly simple:

Start with individual characters. Each character is a token.
Count all adjacent pairs of tokens in the training text.
Merge the most frequent pair into a new token and write down the rule.
Repeat until the vocabulary has the size you want. GPT-2 and GPT-3 stopped after 50,000 merges.

Try it

Train your own BPE tokenizer

Every click finds the most frequent adjacent pair and merges it. Watch the text get shorter and the vocabulary grow. A space is shown as a dot and belongs to the word that follows it.

Task: in text 1, watch “at” appear first, then “th” and “the”, and soon whole words “·cat” and “·the”. In text 3 the token “token” is born very quickly. Notice that the merges have no linguistic meaning. They are only about what repeats often in the text.

The rules are stored in the order they were created. When the tokenizer later gets a new text, it splits it into characters and applies the rules in that same order. That is why the same text always gives the same tokens.

The real GPT-3 tokenizer

GPT-3 uses the same tokenizer as its predecessor GPT-2. It has 50,257 tokens, and that number has a simple origin:

256 bytes + 50,000 merges + 1 special token <|endoftext|> = 50,257 The special token marks the end of a document. During training it separates texts that have nothing to do with each other.

Unlike the toy, GPT-2 does not start with characters but with bytes. Any text can be written in UTF-8 as a sequence of bytes (numbers 0 to 255). A plain English letter is one byte, the Slovak “č” is two bytes, an emoji four. Thanks to this the tokenizer can handle absolutely anything: Chinese, code, emoji. It never meets an unknown character.

Before merging, the text is split into words, numbers and punctuation. A space sticks to the start of the following word. That is why “ world”, including its space, is a single token with ID 995, and “Hello world” is two tokens: [15496, 995].

Try it

The GPT-3 tokenizer right in your browser

This is not an imitation. All 50,000 merge rules of GPT-2 (456 kB) are loaded into your browser and the text is split exactly the way GPT-3 saw it. Type anything. Below each token is its ID. Point at a token to see its bytes and what it was made from.

Task: compare the English and the Slovak example. The Slovak text has 63 characters and 30 tokens, the English one 73 characters and only 17 tokens. Find the tokens marked in red: they are halves of accented letters. Then type your own name and check whether it is a single token.

Why does Slovak do worse? The merge rules were learned from internet text that was almost entirely English. English words therefore often merged into a single token. Slovak word forms were rare, so they stayed chopped up. On top of that, a letter with a háček or an acute accent has two bytes, and sometimes even those two do not merge, so a token is just half a letter.

Why it matters

In English one token is about 4 characters. In Slovak it is often only 2 to 3. The same text in Slovak therefore takes more tokens, more room in the context window and more computation. Using models through an API is billed per token. Newer models have vocabularies of 100,000 to 200,000 tokens and handle other languages better.

From a token number to a vector

The tokenizer turned the text into a list of IDs, for example “ cat” has ID 3797. But you cannot compute anything sensible with such a number. A bigger ID does not mean “more” of anything, it is just a position in the vocabulary. The network needs a whole vector for each token: a list of numbers that express something.

The first step is a one-hot vector: 50,257 zeros and a single one at the position of the given token. When you multiply a matrix by it (as in lecture 1, weights times inputs), every column gets multiplied by zero except one: the column of that token.

W_E · one-hot(3797) = column no. 3797 of W_E = the embedding of “ cat” W_E is the embedding matrix. In practice nothing is multiplied, the computer simply reads the column. The result is the same.

This matrix is called the embedding matrix. In GPT-3 it has 12,288 rows and 50,257 columns. Every token has its own column of 12,288 numbers, and that column is called the token’s embedding. The number 12,288 is called d_model: the width of the model. All of GPT-3, from start to finish, works with vectors of this length.

The embedding matrix W_E in GPT-3. The token’s column is simply picked out. The values on the right are only illustrative.

50,257columns

One for every token in the vocabulary.

12,288numbers per column

The length of every embedding, d_model in GPT-3. The original 2017 Transformer had 512.

617,558,016parameters

50,257 × 12,288. About 0.35 % of GPT-3’s 175 billion parameters.

Where do these numbers come from? At the start of training they are random. They are ordinary network weights, exactly as in lecture 1, and they are learned the same way: by gradient descent and backpropagation. When the model predicts the next token badly, the error flows back all the way to the embeddings of the input tokens and nudges them a little.

Try it

A one-hot vector picks a column

A small embedding matrix: 7 tokens and 11 numbers each. Click a token. The one-hot row has a one only under it, so the multiplication keeps just its column.

Task: compare the columns “king” and “queen”. They differ in almost a single row. I set these values by hand and named the rows to make it readable. In GPT-3 they are learned and the rows have no names.

From the paper Attention Is All You Need (2017)

Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension d_model.

Section 3.4. Vaswani et al., 2017.

Meaning as a direction in space

A vector of two numbers is a point in the plane, or an arrow from zero to that point. A vector of three numbers is an arrow in space. An embedding of 12,288 numbers is an arrow in a space with 12,288 directions. Nobody can picture such a space, but computing in it works exactly as in the plane.

After training something remarkable shows up. Tokens with similar meanings have similar vectors. Nobody programmed this. The model only learned to predict the next token. Words that appear in similar sentences (cat, dog, hamster) must lead to similar predictions, and so they end up close together.

Measuring similarity: the dot product

To compare two vectors we use the dot product. Multiply the numbers in matching positions and add everything up. It is exactly the weighted sum from lecture 1.

a · b = a₁·b₁ + a₂·b₂ + … + a_n·b_n = |a| · |b| · cos(angle) Positive: the arrows point in a similar direction. Zero: they are perpendicular, unrelated. Negative: they point against each other. Cosine similarity is the product divided by the lengths and always lies between −1 and 1.

Try it

The dot product of two arrows

Drag the tips of arrows a and b. On the right the product, the lengths, the angle and the cosine are recomputed.

Task: make the arrows perpendicular and check that the product is zero. Then make arrow a twice as long: the product doubles, but the cosine stays the same. Cosine measures only direction, not length.

Remember the dot product. In lecture 5 you will see that attention, the heart of the transformer, is at its core just a lot of dot products: every token asks which other tokens point “in a similar direction” to its question.

A walk through embedding space

The next demo uses a small hand-built embedding: 40 words, each with 11 numbers. Unlike GPT-3, every axis has a name (gender, age, animal, country…) so you can read what the numbers mean. In a real model meaning is spread across many axes at once and no axis has a simple name.

Try it

Embedding space explorer

Point at a word to see its vector and nearest neighbours. Click two words to compare them with the dot product. The switch selects which two axes are drawn.

Task: compare “king” and “man” (high cosine), then “king” and “Paris” (around zero). For “king” and “queen” it comes out only around 0.5: the gender axis has opposite signs and pulls the dot product down. Switch to the “east × north” view: countries and their capitals sit at the same spot on the map.

The “Overview” view squeezes 11 dimensions into two so that as many differences as possible stay visible (a method called PCA). Every such squeeze loses something. Two words can sit side by side in the picture and actually be far apart. So always look at the numbers too.

Doing arithmetic with words

If meanings are directions, you can compute with them. The difference queen − king is an arrow that goes from “male” to “female”. You find the same arrow between “man” and “woman”. So if you add it to “king”, you should land somewhere near “queen”.

king − man + woman ≈ queen First shown in 2013 by the authors of word2vec (Mikolov et al.). 3Blue1Brown demonstrates it in his video on real embeddings.

The 3Blue1Brown video has more examples. Italy − Germany + Hitler lands close to “Mussolini”: the model apparently built a direction for something like “Italian-ness” and one for “World War II leader”. There is also a “plural” direction: the dot product with it grows steadily for the words “one, two, three, four”. Nobody designed these directions, they emerged on their own during training.

Try it

Word arithmetic: A − B + C = ?

Pick an example or build your own. The solid arrow goes from B to A, the dashed one is the same arrow moved to C. Where it ends, we look for the nearest word. The picture is a slice of the space that contains all four points.

Task: try “Paris − France + Slovakia” and “puppy − dog + cat”. Then invent an example that does not work, such as “dog − man + Paris”. The arithmetic only works when the difference A − B really captures a single property.

An honest remark

In the demo everything works out nicely because I set the axes by hand. In real models it only holds approximately: the result is usually not exactly “queen”, it is just closer to it than other words are (and the input words are excluded from the search). But the idea is right: in embedding space, directions carry meaning.

Where in the text a token stands

Compare two sentences: “It’s not bad, it’s good.” and “It’s not good, it’s bad.” They have exactly the same tokens, just in a different order, and the opposite meaning. Yet the embedding of the token “ good” is the same at position 3 as at position 7. And the transformer processes all tokens at once, not one after another. Without help it would not see the order at all.

The fix is simple. To every embedding we add another vector that says at which position the token stands. In 2017 the authors of the Transformer used waves for this:

PE(pos, 2i) = sin( pos / 10000^{2i / d_model} ) PE(pos, 2i+1) = cos( pos / 10000^{2i / d_model} ) pos is the token’s position, i is the index of a pair of dimensions. Even dimensions get a sine, odd ones a cosine.

Picture a clock with many hands. The first dimensions are the second hand: they spin fast and change a lot at every position. The last dimensions are the hour hand: they only move after hundreds of positions. From the positions of all hands together you can read off exactly which position it is.

Try it

Positional encoding from the formula

Each row is the vector of one position, each column one dimension. The colours are computed with the formula above. Point at a cell to see the calculation. Click to choose positions A and B.

Task: watch the similarity chart in the bottom right. Positions close to A have high similarity, distant ones lower. That is how the model can tell what is near and what is far. Also notice that the columns on the right change very slowly: those are the hour hands.

From the paper Attention Is All You Need (2017)

We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PE_pos+k can be represented as a linear function of PE_pos. We also experimented with using learned positional embeddings instead, and found that the two versions produced nearly identical results.

Section 3.5. The wavelengths form a geometric progression from 2π to 10000 · 2π.

GPT-3 does not use waves. Like GPT-2, it has a second matrix that it learns like everything else: positional embeddings. It has 2,048 rows (one per position) and 12,288 columns, 25,165,824 parameters in total. The input to the first layer is then a plain sum:

input_pos = W_E[token] + W_P[pos] Both parts have 12,288 numbers. The original Transformer also multiplied the embedding by √d_model before adding. 3Blue1Brown’s tally of 175,181,291,520 parameters leaves the positional embeddings out.

Encoder and decoder: three families of transformers

The 2017 Transformer was built for translation, for example from English to German. It had two parts, each with 6 layers:

Encoder

Reads the whole sentence in the source language at once. Every token may look at all the others, forwards and backwards. The result is an “understanding” of the sentence: one vector per token.

Decoder

Writes the translation token by token, left to right. It sees the whole source sentence (through the encoder), but of the translation only what it has already written. The future is masked.

Three families of models grew out of this kit:

Family	Examples	Good for	Who sees whom
Encoder–decoder	Transformer (2017), T5	translation, summarisation	encoder everything, decoder to the left
Encoder-only	BERT (2018)	understanding, search, classifying texts	everyone sees everyone
Decoder-only	GPT, GPT-2, GPT-3, ChatGPT	writing text	each only to the left

Try it

Three families: who may look at whom

Switch between architectures. Click a token (in the diagram or in the grid) and the tokens it may “see” light up in violet. For the translator and for GPT, step through generation with the button.

Task: look at the grid for GPT. It is a triangle: each token sees itself and everything before it, nothing after it. For BERT the grid is full. Why can’t BERT write text? Because during training it always saw the words to the right too, and those do not exist yet while writing.

GPT kept only the decoder of the Transformer. Since there is no encoder, the part through which the decoder looked into the encoder is gone too. What remains is a stack of 96 identical layers in which every token sees only the tokens before it.

Why the decoder won

Thanks to the “left only” mask, every position in the text is a separate training example. From one stretch of 2,048 tokens the model learns 2,048 predictions at once: the next token after the first token, after the first two, after the first three… And the training text needs no labels at all, you just need to have it.

How exactly tokens “talk to each other” is the topic of lecture 5. For today it is enough to know who may listen to whom.

The way out: from a vector back to a token

After 96 layers every position again has a vector of 12,288 numbers. The one at the last position contains everything the model needs to predict the next token. All that is left is to turn it into 50,257 probabilities.

That is the job of the unembedding matrix W_U: 50,257 rows and 12,288 columns. Each row belongs to one token. The final vector is multiplied by every row, giving 50,257 dot products. The more the vector “points towards” a token, the higher its score. These scores are called logits.

logits = W_U · vector → p_i = e^{logit_i / T} / Σ_j e^{logit_j / T} W_U has 50,257 × 12,288 = 617,558,016 parameters. The second part is softmax with temperature T: it turns arbitrary numbers into probabilities between 0 and 1 that add up to 1, keeping their order.

Logits can be any numbers, positive or negative. Softmax first exponentiates them (e^x is always positive and makes big numbers even bigger) and then divides by the sum so they add up to 1. The temperature T controls how sharp the resulting distribution is.

Try it

Logits, softmax and temperature

Eight candidates for the next token. Drag their logits, change the temperature and top-k. The “Sample 100” button shows how often the model would actually pick each token.

Task: at temperature 0.1 the best token gets almost all of the probability. At temperature 2 even “moon” has a chance. Raise the logit of “banana” by 1 (at temperature 1): its probability grows about 2.7 times, because e¹ ≈ 2.7.

Temperature near 0: the model always picks the most likely token. The text is safe but often repetitive.
Temperature 1: the probabilities exactly as the model learned them.
Temperature above 1: unlikely tokens get a chance too. The text is more creative, but also more chaotic.
Top-k and top-p: before sampling, the “tail” of nonsense tokens is thrown away. Top-k keeps the k best, top-p keeps the smallest group of best tokens whose probabilities add up to at least p.

And where is the learning?

During training nothing is sampled. We look at the probability the model gave to the token that actually came next in the text. The error is −log of that probability (cross-entropy). It is the network’s cost from lecture 2, just for words. Backpropagation then sends it back through W_U, all 96 layers, and down to the embeddings.

By the way, the original Transformer and GPT-2 use the same matrix for W_E and W_U (just transposed). It makes sense: a token that goes in as a vector should have the same direction when it is being predicted. 3Blue1Brown counts them separately in his parameter tally.

The context window: 2,048 tokens

GPT-3 processes at most 2,048 tokens at once. This is called the context window. It is a hard limit: the positional embedding matrix has exactly 2,048 rows, so there simply is no vector for position 2,049.

2,048tokens

The maximum input length of GPT-3 (n_ctx in the paper). The question and the answer both have to fit inside.

≈ 1,500English words

About three pages of text. Less fits in Slovak, because Slovak words take more tokens.

0memory between conversations

The weights do not change during a conversation. The model only “remembers” what is currently in the window.

Everything the model “knows” about the conversation has to be in that window. When you chat with a chatbot, the whole conversation so far is sent to it again for every reply. When it gets longer than the window, the oldest parts have to be cut or summarised. That is why a model sometimes “forgets” what you told it at the start.

Newer models have windows of tens of thousands up to millions of tokens. The principle is the same, though: the model sees only what is in the window and predicts one next token from it.

Check what you understood

Six short questions. Nothing is sent anywhere, it is just for you.

1. How many parameters does the GPT-3 embedding matrix have?

2. Why does a Slovak sentence have more tokens than an equally long English one?

3. Two embeddings point in exactly opposite directions. What is their dot product?

4. Why is a positional vector added to the embedding?

5. Which family does GPT-3 belong to?

6. What happens when you lower the temperature almost to zero?

What you take away

A language model is a function: text so far → probabilities of the next token. Writing is a loop: predict, sample, append, repeat.
Tokens are pieces of words found by the BPE algorithm. GPT-3 has 50,257 of them, and Slovak breaks into more tokens than English.
The embedding matrix (12,288 × 50,257) is a table of learned weights. Every token gets its own column: a vector of 12,288 numbers.
Meaning is a direction. Similar tokens have similar vectors, and the dot product measures how similar.
Position is added as another vector: waves in 2017, a learned 2,048 × 12,288 matrix in GPT-3.
GPT is decoder-only: each token sees only what comes before it. At the output, the unembedding gives logits and softmax with a temperature turns them into probabilities.

So now we know how text gets in and how a word gets out. In between are 96 layers that turn isolated embeddings into vectors full of context. The word “key” has to mean something different in a sentence about a lock than in a sentence about music. How that happens is shown by attention in the next lecture.

Lecture four is done

Continue to lecture 5

Sources

T. Brown et al.: Language Models are Few-Shot Learners (GPT-3). 2020.
A. Vaswani et al.: Attention Is All You Need. 2017. Sections 3.1, 3.4 and 3.5.
3Blue1Brown: Transformers, the tech behind LLMs (video, 2024).
A. Radford et al.: Language Models are Unsupervised Multitask Learners (GPT-2, byte-level BPE). 2019.
R. Sennrich, B. Haddow, A. Birch: Neural Machine Translation of Rare Words with Subword Units (BPE). 2016.
T. Mikolov et al.: Efficient Estimation of Word Representations in Vector Space (word2vec). 2013.
J. Devlin et al.: BERT. 2018.
GPT-2 vocabulary and merge rules: openai-community/gpt2 (MIT licence).