The Capstone: Build a Grounded Assistant

Seven lessons ago you learned what a neuron computes. Today you assemble the whole stack into the most practically useful pattern in applied AI right now: an assistant that answers questions from your documents, with citations, on your machine — and, crucially, an evaluation harness that tells you honestly when it fails.

That last clause is the real lesson. Anyone can wire up retrieval in an afternoon (you will, below, in about sixty lines). What separates a toy from a tool — and a safe deployment from a liability — is knowing its failure rate on your questions over your corpus. By the end you'll have both the tool and the number.

What we're building

The complete script: build_rag_assistant.py. Setup is two installs — pip install sentence-transformers numpy, plus optionally pip install anthropic and an API key if you want generated answers rather than retrieval-only. Everything else below walks through the decisions in the code, which is where the learning lives.

Step 0: choose your corpus — carefully

Any folder of .txt or .md files works. Aim for something between twenty and a few hundred documents that you know well — that last property is what makes the evaluation meaningful. Good choices: a society's published guidelines, a course's worth of lecture notes, your own blog or thesis, a set of departmental protocols if and only if they contain no patient data.

Step 1: chunking — the decision everyone underestimates

The script splits each document into windows of ~1,200 characters with 200 characters of overlap between consecutive chunks. Both numbers are arguments you should experiment with, because chunking is where more real-world RAG systems quietly fail than anywhere else.

Why chunk at all? Because retrieval granularity is answer granularity. Embed whole documents and your retriever can only say "the answer is somewhere in this 40-page guideline" — too coarse to ground a specific claim. Chunk too small and you get the orphaned-context problem from Lesson 7: a chunk reading "the dose is 5 mg once daily" retrieves beautifully and means nothing, because the drug name lives in the previous chunk. The overlap is the compromise: each boundary region appears in two chunks, halving the chance that a critical sentence gets severed from its context.

The production-grade refinement, once you've felt the failure: split on structure (section headings, paragraphs) rather than character counts, and prepend each chunk with its document title and section path — "AF Guideline 2025 › Anticoagulation › Perioperative management: …". Ten lines of extra code; dramatic retrieval improvement on hierarchical documents like guidelines. The script keeps character windows for clarity; the upgrade is your first exercise.

Steps 2–4: embed, retrieve, answer

The embedding model runs locally — a small sentence-transformer, ~90 MB, downloads on first run, embeds a few hundred documents in about a minute on a laptop, no data leaving your machine. Retrieval is Lesson 7's twenty lines: normalise, dot product, top-k. Answering assembles the prompt you've already seen, with the two instructions that do the safety work: answer only from the passages and say so if they don't contain the answer.

One design choice worth pausing on: the script prints the retrieved passages before any generated answer — always, in every mode:

Get in the habit of reading that list before reading any generated answer. It is the system's differential diagnosis — and note the two lower-ranked passages: a 2019 warfarin protocol (topically adjacent, wrong drug class) and dental advice (relevant only sometimes). Lesson 7's Figure 3, live in your own terminal. Whether the generator handles them gracefully is exactly what the next step measures.

Step 5: the evaluation — where the capstone earns its name

Now the part that separates this lesson from every RAG tutorial on the internet. Write a file called eval.tsv: one line per test, a question you know the corpus can answer, a tab, and the filename that contains the answer.

Then python build_rag_assistant.py docs/ --eval eval.tsv scores recall@5: for what fraction of your questions did the right document appear in the top five retrieved chunks? This is the retrieval-only metric Lesson 7 argued for — no LLM involved, no fluency to seduce you, just a number.

Two disciplines while you iterate. Don't tune on your test set into oblivion — after a few fix-and-re-run cycles, write ten fresh questions and see if the improvements held (derivation and validation cohorts; you know this dance). And spot-check faithfulness separately: for five questions where retrieval succeeded, read the generated answer against the passages, sentence by sentence. Anything asserted that the passages don't support is Lesson 6's polished hallucination surviving inside a "grounded" system — the single most important thing to catch, and the reason the citation instruction exists.

Where this connects back — and forward

Wire the pieces into Lesson 3's picture and notice what you've built: retrieve() is a tool; the folder is Memory; the answer loop is one turn of the agent loop. Give an agent this script as a callable tool and it will search your corpus iteratively — rephrasing queries, chasing cross-references, synthesising across files — which is exactly the fix Figure 2 prescribes for the hardest failure class. The capstone isn't beside the agent architecture; it's a component of it.

And the series arc is now closed. A neuron (Lesson 1) — trained by hand (2) — stacked into a transformer (4) — trained into a language model (5) — shaped into an assistant (6) — grounded in your documents (7, 8) — wrapped in an agent that acts (3). Eight lessons, and there is no box in any vendor's architecture diagram you can't now open.

What's next

The foundations are laid, which changes the character of what follows: from here the series turns to the topics that keep practitioners honest — evaluating LLM systems properly (beyond today's recall@k), interpretability (what's actually inside the weights you trained), and multimodality (images, and what that means for those of us who look at scans and slides for a living). In whichever order the questions arrive.

Until then: build the thing. Point it at documents you know, write your twenty questions, get your number. Then — this is the graduation exercise — show a colleague the beautiful cited answer and the ranked passage list behind it, and teach them the difference. That's the whole curriculum, paid forward in five minutes.

— Neal