Meaning You Can Search: Embeddings, Retrieval, and Grounding

The assistant we finished in the last chapter has a strange and specific kind of ignorance, and it is worth stating precisely, because the whole of this chapter is a response to it.

It is not that the assistant knows too little. It knows an extraordinary amount — a compressed, blurry impression of a large fraction of everything ever written, up to the day its training data was collected. The problem is threefold and exact. It knows nothing that was written after that day. It knows nothing that was never public — not your hospital's formulary, not the protocol your department agreed last month, not the notes of the patient in front of you. And, as Chapter Six established, when it does not know, it has been trained to produce a confident, well-mannered answer regardless, sometimes complete with a citation that corresponds to no real source. It is a brilliant mind, sealed inside a moment, with a trained disposition to bluff.

You might imagine the fix is to retrain it on the missing knowledge. For facts, this is almost always the wrong tool — we saw why in Chapter Five. Pretraining is a months-long, multi-million-dollar industrial process, and the lighter fine-tuning of Chapter Six relocates a model's defaults, not its reliably-recallable knowledge. What we actually want is something far more surgical: at the very moment a question is asked, reach into a body of documents we control, find the passages that bear on it, and lay them in front of the model as part of its context — so that when the model does its Chapter Four forward pass, the evidence is already on the page. The model then does what it has always done, continue the text, except the text now contains the facts.

That is retrieval-augmented generation — RAG — and it rests on a single idea of real beauty, one we have already met in miniature and now scale up: that meaning itself can be given coordinates.

Part I — The same geometry, scaled up

Cast your mind back to Chapter Two. We took each token and gave it a vector — a location in a high-dimensional space arranged so that tokens with similar meaning sit near each other, and directions encode relationships. I promised, at the time, that this idea would return, scaled up from single tokens to whole documents. Here it is.

An embedding model — itself a transformer, trained for precisely this one job — reads an entire sentence, paragraph, or passage and outputs a single vector that represents the meaning of the whole thing. And it is trained so that passages with similar meaning land near each other in the space. Not similar wording — similar meaning. This distinction is the entire point, so let me make it concrete. A good embedding model places "the patient developed an itchy rash after her first dose of amoxicillin" close to "penicillin allergy — urticaria documented," and far from "the rash resolved after the statin was stopped" — even though, word for word, the sentence about the statin shares more vocabulary with the first than the allergy note does. Meaning, not surface, decides proximity.

How does it learn to do that? By training on contrast, at scale — shown millions of pairs known to be related (a question and its answer, a title and its abstract, two paraphrases of one sentence) and nudged, by the same gradient descent of Chapter Five, to pull related pairs together and push unrelated ones apart, until distance in the space means dissimilarity of meaning.

Figure 1. Chapter Two's geometry, now at the scale of whole passages. Note the decisive win over old-fashioned keyword search: the question about co-amoxiclav shares no words with the penicillin-allergy note, and the word "rash" sits in the lipid cluster — yet the query still lands where it should, because the embedding model matches on meaning. A keyword search would have missed the most important note in the chart.

💡 The reduction that powers everything. An embedding turns a piece of text into a fixed address in a space where distance means dissimilarity of meaning. Once text is geometry, the vague task "find the documents relevant to this question" becomes the crisp, ancient, computationally-cheap task "find the nearest points to this point." That single reduction — meaning to coordinates, relevance to distance — is what makes semantic search, retrieval, and the memory of an AI system all possible with the same machinery.

Two practical notes before we build with it. Similarity between two vectors is usually measured by the cosine of the angle between them — how nearly they point the same way, ignoring their lengths. And searching for nearest neighbours among millions of vectors is made fast by specialised indexes (the "vector databases" you may have heard named are, at heart, exactly this index plus bookkeeping). At the scale of a single clinic — thousands of documents — you do not even need them; comparing a query against every stored vector takes milliseconds.

Part II — The pipeline: two lanes meeting at the context window

With embeddings in hand, retrieval-augmented generation is two pipelines that meet at a single point — the model's context window from Chapter Four.

The first lane is done once, ahead of time, and refreshed on a schedule. Take your documents — guidelines, protocols, papers, notes. Cut them into passages of a manageable size (this "chunking" step is quietly one of the most consequential choices in the whole system, and we will see why in Part V). Run every chunk through the embedding model to get its vector. Store the vectors, alongside the original text and a note of where each came from, in an index. That is the ingestion lane: a library, converted into searchable geometry.

The second lane runs every time a question is asked. Embed the question with the same model, into the same space. Find its nearest neighbours in the index — the handful of chunks whose meaning sits closest to the question. Then assemble a prompt that places those retrieved passages, plus the question, plus a clear instruction, into the model's context window — and let the model answer.

Figure 2. The full pipeline. The vital thing to notice: the model itself is unchanged — the same weights from Chapter Six, doing the same forward pass from Chapter Four. Every piece of new engineering lives outside the model, in the machinery that decides what gets placed into its context window. Fresh knowledge is one re-index away, no retraining involved.

Sit with the modesty of that architecture for a moment, because it is easy to miss how much it buys. We did not touch the model. We built a search engine over documents we control, and we arranged for its results to be handed to the model as reading material at the instant of the question. The two instructions in that context window — answer only from these passages and say so if they don't contain the answer — are what do the safety work, converting a bluffing generalist into a grounded specialist for the length of one reply.

Part III — Why this beats retraining, and heals Chapter Six's wound

It is worth being crisp about why retrieval, and not fine-tuning, is the right answer to "make the model know our material" — because the alternative is sold constantly, and because one of retrieval's advantages is precisely the wound we left open at the end of the last chapter.

Figure 3. Why retrieval is the default answer for knowledge. The starred advantage is the one that matters most here: at the end of Chapter Six we saw that "the model said so" is not provenance, because nothing in training ties an assertion to a source. Retrieval is what finally ties them — the model's answer can carry real references to real passages a human can open and verify.

Read those four advantages against the alternative and the case is close to decisive for factual knowledge. A fine-tune bakes yesterday's guideline permanently into the weights, available to every user, unciteable, and impossible to revoke. An index is a living document set: update it, restrict it per user, and — because the answer is drawn from named passages — audit it. And when the right passage simply is not there, a well-instructed model asked to answer only from what it was given will decline far more reliably than one asked to consult its own parametric memory, where, as we know from Chapter Five, something plausible-sounding is always available.

Part IV — Where retrieval fails, and how to catch it

Retrieval is the best answer we have to grounding, which is exactly why you must know its failure modes cold. They fall into two families, and a clinician's instinct for how diagnostic cascades fail will serve you well here.

The retrieval can fail — the right passage exists in the index but does not come back. Sometimes the cause is bad chunking: the answer was split across two chunks, or a chunk was severed from the context that gave it meaning ("the dose is 5 mg" — of what?). Sometimes it is a vocabulary the embedding model handles poorly, or a question whose answer requires synthesising across many documents when only a handful of chunks were retrieved. And sometimes it is a subtler trap that deserves its own picture, because it is the one I test for first in any clinical retrieval system.

Figure 4. The failure I watch for first. Contradictory guidance on the same topic — current and superseded, adult and paediatric, "do" and "don't" — embeds close together, because topical similarity is exactly what the geometry captures. Currency, population, and polarity must be handled by metadata, index hygiene, and the model's reading of the retrieved text — never by the vector search, which is blind to all three.

The generation can fail even when retrieval succeeded — the right passages arrive and the model still gets it wrong. It may blend the retrieved facts with its own parametric memory, answering from the blur of pretraining rather than the document in front of it. It may over-trust a passage that matched on topic but does not actually apply. Or the crucial passage may sit in the middle of a long, stuffed context, where models demonstrably attend least — recall from Chapter Three that attention can reach anything in the window, but Chapter Six's training instilled habits about where it tends to look.

The discipline that follows is the same one we apply to any diagnostic cascade: measure the stages separately. Score the retrieval on its own — for a set of test questions, did the right passages come back at all? This is checkable without any model, just by inspecting what the search returned. Then, separately, score the generation given correct passages — was the answer faithful to them, and did it cite? A single end-to-end accuracy number confounds the two exactly the way "overall mortality" confounds case-mix with quality of care, and is about as useful for finding out what to fix.

⚕️ Two things to settle before any clinical deployment. First: an embedding is derived from the text it represents and can be partially inverted — so a vector index built from clinical notes is itself protected health information, with the same storage, access, and audit obligations as the source records. A folder of vectors is not an anonymous by-product; treat it as the chart. Second: when a retrieval product demos a beautiful cited answer, ask to see the passages it retrieved but was not shown — the ones ranked just below the cutoff. Whether the truth was sitting one rank below the line is the single most informative thing you can learn about the system's margin of safety.

Part V — A window, not yet a door

Stand back and see what we have added. The sealed assistant of Chapter Six can now reach outside itself. Point it at a body of documents you control, and at the moment of a question it searches them by meaning, pulls the relevant passages into its context, and answers from them — with citations a human can open and check. The bluffing generalist has become, for the length of a reply, a grounded specialist that shows its working. The wound we left at the end of the last chapter — fluent confidence with no provenance — is, at last, dressed.

And yet notice the shape of what we have built, because it points straight at the final chapter. Everything in this chapter is reactive and single-shot. A question comes in; we do one retrieval; we produce one answer. The retriever is a tool — a thing the system can use to reach beyond its own weights — but it is a tool wielded once, on a fixed schedule, by machinery outside the model. The model itself did not decide to search. It did not look at what came back and think "that's not enough, let me search again with a better query." It did not, having found a guideline, go on to check the patient's notes, and then a drug database, chaining one lookup into the next. It answered once and stopped.

But a hard question in the real world is rarely one lookup. It is a small investigation — search, read, refine the search, cross-reference, decide what to do next. What if the model could do that? What if, instead of handing it a single retrieval, we gave it a set of tools — search, a calculator, a way to read a file, a way to take an action — and let it decide, step by step, which to use and when, looping until the task was actually done?

That is the leap from a model that answers to a system that acts. It is where retrieval becomes one tool among many, where the single-shot reply becomes a loop, and where the language model stops being an oracle you consult and becomes an agent that works on your behalf. Building it — and understanding both its power and the new care it demands — is Chapter Eight, the last piece of the machine.

— Neal