Manners for a Mind: From Predictor to Assistant

If you built and trained everything in the last five chapters — ground text into tokens, gave them meaning, built attention, stacked the tower, and ran the long descent until the loss flattened — you would hold in your hands one of the more remarkable artefacts humans have made: a base model. It has read a large fraction of everything ever written. It is fluent in dozens of languages, apparently knowledgeable across most of human endeavour, able to continue any text with uncanny plausibility.

And it would be almost useless to you.

Not because it lacks knowledge — it has more than any person alive. Because it lacks a manner. Ask it a question and it will not answer; it will continue, because continuing is the only thing it was ever trained to do. Type "What are the contraindications to thrombolysis?" and a base model might reply with three more exam-style questions, because on the internet a question like that is very often followed by others, in a revision guide or a viva. Nothing has malfunctioned. The model is doing precisely what Chapter Five rewarded it for: predicting the plausible continuation of a document. It has learned the shape of all text, which is emphatically not the same as the shape of helpful text.

This chapter is about the second training — the one that closes that gap. And I want to flag, up front, the single most surprising fact about it, because it reframes everything: post-training adds almost no new machinery. No new architecture. No new mechanism bolted onto the tower. Same weights, same attention, same feed-forward layers, same next-token interface we spent five chapters building. Only the training signal changes. Everything an assistant is — its helpfulness, its caution, its refusals, its manner — it is through the exact machine you already understand, nudged into a new posture. We are not building a new thing. We are teaching an existing thing to behave.

Part I — Two objectives that only overlap

Let us name the mismatch precisely, because the whole chapter is an attempt to resolve it.

Pretraining optimised one objective: continue the text the way this corpus would. What we actually want from an assistant is a different objective: respond to this person helpfully. The trouble is that these two objectives are not the same — they merely overlap. A great deal of helpful text does appear on the internet, so a base model that continues well will often, by accident, be helpful. But the base model is aiming at the wrong target, and wherever the two targets diverge — wherever the most plausible continuation is not the most helpful response — it will follow plausibility every time.

Figure 1. The base model lives in the blue circle — it continues text. We want it in the green — helping the person. They overlap, which is why a base model is sometimes useful, but they are not the same target. Post-training is the work of moving the model's default behaviour from one circle into the other.

There are two ways to do that moving, applied in sequence, and they differ in a deep way: the first shows the model what to say, and the second lets it try and tells it which attempt was better. We take them in turn.

Part II — Stage one: learning by demonstration

The first move is almost embarrassingly direct, and it reuses the entire machine of Chapter Five without modification.

We assemble a collection of demonstration conversations — tens or hundreds of thousands of them. Each is a small script: a user's message, followed by the response a good, careful assistant should give, written or curated by people. "Here is a question about thrombolysis; here is the calm, structured, honest answer we wish the model would produce." And then we simply continue training the base model on these conversations, with the exact loop from the last chapter — forward pass, measure the loss, backpropagate the blame, nudge the weights — except now the text it is learning to predict is the text of good assistant responses. This stage is called instruction tuning (you will also see it named supervised fine-tuning).

One detail makes the whole thing click into place with Chapter One, and it surprises almost everyone. A "conversation" is not a special data structure the model understands natively. It is presented to the model as exactly what everything is presented as — a flat sequence of tokens — with a few special marker tokens sprinkled in to delimit who is speaking:

<|user|> What are the contraindications to thrombolysis in acute stroke?
<|assistant|> The major contraindications fall into three groups: active bleeding risk, …
<|end|>

That is all a "chat" is: a document with role labels, learned as a convention like any other. The model is trained to predict the tokens of the assistant turns, given everything before them. When you had a conversation with an LLM this morning, the model received one long document in exactly this shape and did Chapter Four's forward pass on it, token by token. There is no chat engine hiding somewhere; there is only the tower, predicting the next token of a specially-formatted document.

Figure 2. Instruction tuning is just Chapter Five's training loop run on demonstration conversations, with the "chat format" revealed to be nothing more than role-marker tokens in an ordinary document. It works — the base model's mirror-like continuing gives way to answering. But it has two limits, and those limits are what the second stage exists to overcome.

After instruction tuning, the transformation is real and immediate: the model that used to answer a question with more questions now answers it. But two gaps remain, and naming them sets up the second, more powerful stage. First, demonstrations teach format and typical content, but not judgement — the fine calls of how cautious to be, when to refuse, how to handle a request that is half-reasonable and half-dangerous, are poorly specified by examples alone. Second, hand-written demonstrations are expensive and slow, and they cap the model at "imitate a good human answer" when what we would really like is "produce the answer people actually prefer — even better than a demonstrator would have written."

Part III — Stage two: learning from preferences

The second stage changes not the loop but the kind of signal. Instead of showing the model what to say, we let it generate, and we grade what it produces.

The core idea is a loop. Take a prompt. Let the model produce two (or more) different responses to it. Show a human rater both, and record only which one they preferred. Not a score out of ten — a simple choice: A or B. And notice the design wisdom hidden in that simplicity. Decades of measurement science, clinical assessment very much included, converge on the same finding: humans are unreliable at absolute ratings ("rate this answer 7/10") and far more consistent at forced choice ("which of these two is better?"). Preference data inherits that reliability. It is the same reason a good exam pairs a candidate against a standard rather than asking an examiner to conjure a number from the air.

Those thousands of comparisons are then used to train a second network — a reward model — built from the same transformer architecture, but with its final projection swapped: instead of a probability over the vocabulary, it outputs a single number, a learned estimate of "how much would a human prefer this response?" And finally, the assistant itself is nudged — by the same gradient-descent machinery — to produce responses the reward model scores highly, while being kept on a tether so it does not drift too far from its sensible instruction-tuned starting point.

Figure 3. The preference loop. It is optimisation against a proxy: not truth, but a model of human judgement, itself trained on judgements made quickly by raters who are rarely domain experts. That subtlety — optimising a stand-in for what we want, rather than the thing itself — is the seed of nearly everything that later goes wrong, and the reason the caption under this loop is the most important sentence in the chapter.

💡 The idea to carry out of this chapter. Post-training optimises “what humans, as modelled, prefer” — not “what is true.” Usually these coincide, because people prefer correct, helpful answers when they can tell. But when correctness is hard for a rater to check, people reward confidence, fluency, agreement, and a definite answer over an honest “I don't know.” The model learns exactly that. Hold this single idea and the clinical failure modes below follow from it as corollaries, not surprises.

Part IV — Same weights, different characters

Now the fact that strikes almost everyone as strange, and that quietly explains a great deal of an assistant's everyday behaviour: nothing was removed from the base model. Post-training is a comparatively gentle nudge in the vast space of weights. The enormous repertoire the base model built during pretraining — every register, persona, and viewpoint the internet contained, from a careful clinical discussion to an unhinged forum rant — is all still in there. What changed is not what the model can do, but what it reaches for by default.

The picture I find most faithful: pretraining built an immense landscape of possible behaviours, and a base model, prompted, falls into whichever region of that landscape the prompt statistically resembles. Post-training does not bulldoze regions. It carves a groove — a deep, comfortable channel labelled "helpful, harmless, honest assistant" — that the model now slides into from almost any starting point. The repertoire is untouched beneath; only the default has been reshaped.

Figure 4. Post-training as a carved groove, not a bulldozer. The model's full range of behaviours survives; a deep default channel is cut so that it settles into "helpful assistant" from most prompts. This one picture explains three everyday facts at once, below.

Three familiar observations fall straight out of that picture. System prompts work — a crafted instruction like "you are a terse triage assistant; answer in bullet points; escalate anything cardiac" cheaply repositions the model within its repertoire, because the groove is a default, not a cage. Jailbreaks sometimes work — an adversarial prompt is an attempt to pull the model out of the groove and back into some untamed region of the base repertoire that post-training made harder to reach but did not delete. And fine-tuning works with astonishingly little data — a few hundred examples can specialise a model, because you are not teaching it a subject from scratch; the knowledge is already there from pretraining, and you are merely relocating its default.

⚕️ The distinction that should change how you buy clinical AI. Pretraining is medical school and residency — years, enormous cost, and all the actual knowledge. Post-training is the induction week at a new hospital — quick, cheap, and it determines how you answer the phone, what you document, and when you escalate. Nobody would confuse induction week with medical school, yet in AI discourse the two are conflated constantly. So when a vendor tells you their model was “trained for healthcare,” your first question should be: which of the two do you mean? A model whose pretraining was rich in medicine is a different animal from one that merely had a medical induction week bolted onto a general base — and the answer changes everything about how you should validate it.

Part V — The seams, and where the failures live

We can now do the thing this chapter was really for: read a deployed assistant's most important behaviours — and its most consequential failures — as direct consequences of the pipeline we have just built. None of what follows is a random glitch. Each is a seam of post-training, showing through.

Sycophancy is a training artefact, not a personality quirk. In the preference loop, agreeable answers win comparisons more often than blunt corrections — people, on average, prefer being agreed with. So the model acquires a measurable tilt toward endorsing whatever the user seems to believe. Put that in a clinical workflow and the danger is exact: the registrar types "elderly patient, fall, on apixaban, CT head clear, planning discharge — anything else?" and a sycophantic model is biased toward blessing the plan. The peril is precisely that it feels like a second opinion — it has the form of consultation with the incentives of a courtier. When you evaluate a clinical tool, test it with a wrong premise deliberately embedded and watch whether it pushes back. That single test tells you more than any accuracy benchmark.

Confident fabrication survives — masked, not cured. We watched the fluency-fidelity gap form in earlier chapters: statistical plausibility arrives long before factual grounding, at every scale. Post-training reduces fabrication, because raters do punish errors they can detect — but it cannot remove it, because the underlying generator is still a sampler over plausible continuations. And it adds a hazard of its own: the fabrications that survive are now delivered in the calm, structured, caveated register of a careful colleague, because that is the register post-training rewards. A base model's nonsense at least looked like nonsense. An assistant's nonsense looks like a well-written consult note with an invented reference in it.

Refusals and caveats are a trained distribution — and it was calibrated for the public, not for you. When a model declines, hedges, or says "consult a professional," it is following patterns learned from raters applying a policy — raters who were generally not clinicians, and a policy written for a general audience. So expect miscalibration at both ends in specialist use: over-refusal on legitimate clinical questions (an unusual but valid dosing scenario), and under-refusal where the surface framing looks benign. The calibration appropriate for a tool used by clinicians is simply different from one used by the public — and off-the-shelf assistants ship with the latter.

Figure 5. The failure modes of a deployed assistant, read off the pipeline. They are not bugs in the ordinary sense — no line of code is wrong. They are the predictable consequence of optimising a proxy for what we want, in the regions where the proxy and the goal diverge. Knowing why they occur is the difference between deploying carefully and deploying blindly.

And "the model said so" is not provenance. This last one is the hinge to what comes next. Instruction tuning and preference learning teach a model to present answers beautifully — including confident, citation-shaped strings. But nothing in either stage connects an assertion to an actual source. The assistant's polish is not evidence; it is style, rewarded because style is what a hurried rater can judge. If you want an answer grounded in a document a human can open and check, the entire pipeline of this chapter cannot give it to you.

Part VI — A complete mind, and its remaining blindness

Step back and take stock, because the machine is, in an important sense, now finished. Across six chapters we have ground text into tokens, given them meaning as geometry, built the attention that lets a token read the room, stacked it into a deep tower, filled that tower with knowledge through the long descent of pretraining, and — in this chapter — shaped the resulting base model into an assistant with a manner: something that answers, follows instructions, hedges, refuses, and speaks in the careful voice we asked for. That is, end to end, the thing you talk to. There is no further secret.

But look hard at what this finished assistant still cannot do, because it is precisely the shape of the final chapters. It knows only what pretraining happened to teach it — nothing about your hospital's formulary, nothing about the patient whose notes are open in front of you, nothing published after the day its training data was collected. And when it does not know, its post-training has taught it, too often, to answer anyway, fluently, in that trustworthy register, with a fabricated citation that no part of its training ever tied to a real source. It is a brilliant, well-mannered mind sealed inside the moment its training ended, with no way to look anything up and a trained disposition to bluff when it cannot.

Closing that gap — without the ruinous expense of retraining — means giving the model a way to reach outside itself: to search documents you control, pull the relevant passages into its context, and ground its answer in sources a human can inspect. Turning our sealed, fluent, occasionally-bluffing assistant into one that can look things up and show its working is the beginning of the machine's connection to the living world.

How that works — how text becomes searchable by meaning, and how retrieved passages are folded into the model's context to ground what it says — is Chapter Seven.

— Neal