Inside the Transformer

At the end of the last lesson I promised we'd close the gap in the middle of our stack. At the bottom (Lessons 1–2): neurons, weights, gradient descent. At the top (Lesson 3): the agent loop, tools, permissions. Between them sits the machine that does the actual thinking — the thing that receives a transcript and produces, of all the words it could say next, the right one.

That machine is the transformer, and it has been the architecture behind essentially every frontier language model since 2017. This lesson explains it the way I wish someone had explained it to me: not as a wall of matrix algebra, but as a small number of design decisions, each solving a specific problem — ending with working PyTorch code for a complete miniature GPT that you can read in one sitting.

One job: the next token

Strip away the chat interface and an LLM has exactly one skill. Given a sequence of tokens, it outputs a probability for every token in its vocabulary being the next one. That's it. That's the whole job.

Everything else is repetition. To generate a paragraph, the system samples one token from that probability distribution, appends it to the sequence, and asks again. And again. A thousand-word answer is a thousand spins of this wheel:

Hold on to this framing, because it demystifies half the folklore about LLMs. The model has no plan, no draft, no lookahead buffer. Any apparent long-range structure in its output — an argument that builds, code that compiles, a differential that narrows — must somehow be computed fresh at every step, from nothing but the transcript so far. The rest of this lesson is about the machinery that makes that possible.

From text to numbers

Neural networks eat vectors, not words. So before anything interesting happens, text goes through two conversions.

Tokenisation. The raw string is chopped into tokens — chunks from a fixed vocabulary, typically 50,000–200,000 entries learned from data. Common words get their own token; rarer words are assembled from pieces (hyponatraemia might become hypo|nat|ra|emia). This is why LLMs are notoriously shaky at counting letters: the model never sees letters, only chunk IDs.

Embedding. Each token ID indexes into a big lookup table and retrieves a learned vector — several hundred to several thousand numbers. These embeddings are learned during training, and they end up encoding meaning as geometry: tokens used in similar contexts drift toward each other in the vector space. You met this idea in Lesson 1; here it's the front door of the whole model.

One more ingredient: the model needs to know where each token sits, because "dog bites man" and "man bites dog" contain the same tokens. So a position signal is mixed into each embedding — in the simplest scheme, a second lookup table indexed by position 0, 1, 2, …

The breakthrough: attention

Here's the problem the middle stack has to solve. Consider the sequence:

"the drug lowered blood pressure because it …"

To predict what follows "it", the model must know what "it" refers to — and that information lives four tokens back, in "drug". Meaning in language is relational: a token's true significance depends on other tokens, sometimes hundreds of words away, in patterns that change from sentence to sentence.

Older architectures handled this badly. Recurrent networks read left to right, compressing everything seen so far into one fixed-size summary vector — by the time you're 500 tokens in, the details of token 12 have been squeezed to mush. The field needed a mechanism where any token could consult any earlier token directly, with the relevance decided dynamically by content.

That mechanism is attention, and the cleanest way to understand it is as a soft database lookup that every token performs simultaneously.

Each token's vector is projected into three roles:

• a query — "here's what I'm looking for"
• a key — "here's what I contain, as an advertisement"
• a value — "here's what I'll hand over if you pick me"

Every token's query is compared (dot product) against every earlier token's key. Good matches score high. The scores pass through a softmax to become weights summing to 1, and each token receives the weighted average of the earlier tokens' values. The token "it" emits a query shaped roughly like "recent singular noun, thing that can act"; the key of "drug" matches it far better than the key of "because"; so the value flowing back to "it" is dominated by drug-ness.

Two properties make this the breakthrough it was.

It's content-addressed, not position-addressed. The lookup asks "who matches this query?", not "who is 4 slots back?". The same weights machinery handles pronoun resolution, subject–verb agreement, matching a closing bracket in code, and retrieving a fact stated three paragraphs ago — without being told which of those tasks it's doing.

It's completely parallel. Every token computes its lookup simultaneously — one big matrix multiplication, no left-to-right crawl. This is what let transformers train on orders of magnitude more data than recurrent networks: the architecture finally matched what GPUs are good at.

In code, the whole mechanism is about a dozen lines. This runs as written:

import torch
import torch.nn as nn
import torch.nn.functional as F

class AttentionHead(nn.Module):
    def __init__(self, d_model, d_head):
        super().__init__()
        self.q = nn.Linear(d_model, d_head, bias=False)
        self.k = nn.Linear(d_model, d_head, bias=False)
        self.v = nn.Linear(d_model, d_head, bias=False)

    def forward(self, x):                       # x: (batch, tokens, d_model)
        B, T, _ = x.shape
        q, k, v = self.q(x), self.k(x), self.v(x)
        scores = q @ k.transpose(-2, -1) / k.shape[-1] ** 0.5   # (B, T, T)
        mask = torch.tril(torch.ones(T, T, device=x.device)).bool()
        scores = scores.masked_fill(~mask, float("-inf"))       # no peeking ahead
        weights = F.softmax(scores, dim=-1)     # each row sums to 1
        return weights @ v                      # weighted mix of values

That masked_fill line deserves a highlight: it blanks out the upper triangle of the score matrix so that no token can attend to tokens after itself. During training the model predicts every position's next token simultaneously, and the mask is what stops it cheating by reading the answer. This is the causal mask, and it's why this family of models is called autoregressive.

Heads, MLPs, and the residual stream

A real transformer block adds three refinements to the bare mechanism above.

Multiple heads. Instead of one attention operation with big vectors, run 8–100 smaller ones in parallel — each with its own Q/K/V projections — and concatenate the results. Each head is free to specialise: trained models reliably develop heads for syntax, heads that track quoted speech, heads that find the previous occurrence of the current token, heads for bracket matching in code. Interpretability research — the field that dissects trained models — finds these specialisations without being told to look for them, the way histology reveals cell types.

A per-token MLP. Attention moves information between tokens, but does little computation on it. So each block follows attention with a small two-layer network — the plain feed-forward kind from Lesson 1 — applied to each token independently. The rhythm of a block is: communicate, then compute. Gather what you need from the ward, then go away and think about it. Notably, the MLP is where most of a model's parameters live — roughly two-thirds — and current evidence suggests it acts substantially as the model's learned key-value store of facts.

The residual stream. Each sub-layer's output is added to its input, never substituted for it. You can picture each token as owning a running vector — a chart, if you like — that flows up through the stack, with every attention head and MLP reading from it and writing small annotations back onto it:

(The "normalised" in Figure 4 is layer normalisation — each read is rescaled to a standard spread before use. It's the same statistical hygiene as standardising lab values before feeding them to a risk model, and it matters for training stability, not for understanding.)

What a context window physically is

Now we can give a concrete answer to a question I promised in Lesson 3, where we saw the agent loop "compact" its transcript when it grew too long.

The context window is the maximum number of tokens the model can process in one forward pass. It is a physical quantity, set by two things. First, the position signal: the model has only learned to represent positions up to some maximum. Second — and this is the one that costs money — the attention score matrix in every head of every block has one entry per pair of tokens. Double the sequence length and you quadruple the attention computation, and quadruple the memory holding those keys and values.

That quadratic term is why context windows are a headline specification, why long conversations get slow and expensive, and why the agent loop from Lesson 3 summarises old turns rather than keeping everything verbatim. When your coding agent "compacts" the transcript, it is managing exactly this budget.

It also explains something subtler: within its window, the model's recall is direct — attention can reach token 12 from token 5,000 in one hop, with no degradation by distance. Nothing like human memory decay applies inside the window; everything is equally available if some attention head chooses to look at it. Outside the window, recall is exactly zero. The cliff is absolute — which is why agents need the memory files we met in Lesson 3.

A complete tiny GPT

Time to keep the promise: a full GPT-style model, small enough to read in one sitting. This is a real, runnable PyTorch module — embeddings, blocks, the lot:

import torch
import torch.nn as nn

class Block(nn.Module):
    """Communicate (attention), then compute (MLP) — with residual adds."""
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.ln1  = nn.LayerNorm(d_model)
        self.attn = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
        self.ln2  = nn.LayerNorm(d_model)
        self.mlp  = nn.Sequential(
            nn.Linear(d_model, 4 * d_model),
            nn.GELU(),
            nn.Linear(4 * d_model, d_model),
        )

    def forward(self, x):
        T = x.shape[1]
        causal = torch.triu(torch.ones(T, T, device=x.device), 1).bool()
        h = self.ln1(x)
        attn_out, _ = self.attn(h, h, h, attn_mask=causal)  # no peeking ahead
        x = x + attn_out                # write attention result onto the stream
        x = x + self.mlp(self.ln2(x))   # write MLP result onto the stream
        return x

class TinyGPT(nn.Module):
    def __init__(self, vocab_size, d_model=128, n_heads=4, n_layers=4, ctx=256):
        super().__init__()
        self.tok_emb = nn.Embedding(vocab_size, d_model)   # what each token means
        self.pos_emb = nn.Embedding(ctx, d_model)          # where each token sits
        self.blocks  = nn.Sequential(*[Block(d_model, n_heads)
                                       for _ in range(n_layers)])
        self.ln_out  = nn.LayerNorm(d_model)
        self.head    = nn.Linear(d_model, vocab_size)      # scores over vocabulary

    def forward(self, idx):             # idx: (batch, tokens) of integer IDs
        B, T = idx.shape
        pos = torch.arange(T, device=idx.device)
        x = self.tok_emb(idx) + self.pos_emb(pos)          # Figure 2, stages 2–3
        x = self.blocks(x)                                 # Figure 2, stage 4
        return self.head(self.ln_out(x))                   # logits: (B, T, vocab)

Sixty lines, and it is not a toy in any structural sense. Scale d_model to 12,288, n_heads to 96, n_layers to 96, ctx into the hundreds of thousands, train it on a large slice of the internet, and you have sketched a frontier base model. The recipe you'd use to train it is the one you already own: cross-entropy loss on next-token prediction, backpropagation, gradient descent — Lessons 1 and 2, at industrial scale.

Line up the pieces against Figure 2 and notice how little is left unexplained: tok_emb and pos_emb are the lookup tables; each Block is Figure 4; the causal mask is Figure 3's "earlier tokens only" rule; head produces the scores that softmax turns into Figure 1's probability bars.

From next-token predictor to assistant

A short bridge, because the gap puzzles many newcomers: how does "predict the next token of internet text" become the helpful assistant in your terminal?

In stages. Pretraining produces the raw predictor above — call it a base model. Prompt a base model with a question and it may answer, or may continue with nine more questions, because that's a plausible continuation of text containing a question. Post-training then reshapes the distribution: the model is further trained on demonstration conversations, and refined against feedback signals, until "continue the transcript" and "respond helpfully" become the same thing. The architecture never changes — same blocks, same attention — only the weights move. The chat format itself is just more tokens: special markers delimiting who said what, exactly like the transcript the agent loop of Lesson 3 maintains.

That's the full stack, connected: gradient descent (Lessons 1–2) trains the transformer (this lesson), whose next-token interface is wrapped by the agent loop (Lesson 3) — turtles all the way down, except every turtle is now one you've met.

What's next

The obvious move — and the next lesson — is to stop reading and train one. We'll take the TinyGPT above, feed it a corpus small enough for a laptop, write the training loop with our own hands, and watch generated text evolve from noise, to word-shaped noise, to sentences. There is no better calibration for your intuitions about what these models are than watching one condense out of randomness in front of you.

Until then, a self-test: next time you use an LLM, watch the streamed reply and try to see it as Figure 1 — thousands of forward passes, each one Figure 2, each block Figure 4, every token consulting the transcript by Figure 3. Once you can hold that picture, the word "transformer" stops being a brand name and becomes a machine you could sketch on a napkin.

— Neal