Deep Learning Intermediate–Advanced ⏱ 30 min post-traininginstruction tuningRLHFfine-tuningalignmentLLMbase modelsycophancyclinical AIdeep learning

From Predictor to Assistant

Published 2026-07-11 — Dr Neal Aggarwal
📚 Lessons series — #6. This lesson assumes you've met the transformer (Lesson 4) and — ideally — trained one (Lesson 5). It also closes a loop opened in Lesson 3, where an assistant sat at the centre of the agent loop and we took its helpfulness on faith. The full curriculum lives on the Lessons page.

From Predictor to Assistant

If you ran Lesson 5's script, you now own something valuable: a genuinely unhelpful language model. Ask your TinyGPT a question and it will not answer. It will continue — with more anatomy, more Edwardian prose, more of whatever statistical neighbourhood your prompt landed it in. It is a mirror of its corpus, and mirrors don't answer questions.

Yet the models you use daily — the one powering the coding agent from Lesson 3, the one drafting your clinic letters — began life as exactly this kind of mirror, scaled up. Somewhere between "raw predictor trained on the internet" and "assistant that answers, follows instructions, and declines to help with the dangerous stuff," something happened. This lesson is about that something: post-training. It is the least understood stage of the pipeline among otherwise well-informed people, and — I'll argue by the end — the stage whose fingerprints are on the specific failure modes that matter most in clinical deployment.

The problem, stated honestly

Recall the punchline of Lesson 5: a base model completes text according to the statistics of its training distribution. Feed a big base model this prompt:

prompt: What are the contraindications to thrombolysis in acute stroke?

a base model, plausibly: What is the therapeutic window for alteplase? What imaging is required before treatment? These are among the questions most commonly asked by candidates preparing for the written examination…

Nothing malfunctioned. On the internet, a question is very often followed by more questions — exam banks, FAQ pages, forum threads. The base model is doing precisely what we trained it to do in Lesson 5: continue the document. The problem is that "continue the document" and "help the person" are different objectives that merely overlap. Post-training exists to close that gap — to take a model that can complete anything and shape it into one that reliably does the one thing you actually want.

The deepest fact about post-training, and the source of most of its surprises, is how little machinery it adds. No new architecture. No new mechanism bolted onto the transformer. The same weights, the same attention, the same next-token interface — only the training signal changes. Everything an assistant is, it is through the mechanism you already understand.

Stage one: teaching the format by example

The first move is embarrassingly direct: collect tens or hundreds of thousands of demonstration conversations — a user message, followed by the response a good assistant should give, written or curated by humans — and continue training the base model on them with exactly Lesson 5's loop. Batch, forward, cross-entropy, backward, step. This is instruction tuning (you'll also see supervised fine-tuning).

One detail makes it click into place with what you already know. A "conversation" is presented to the model as — of course — a flat token sequence, with special marker tokens delimiting the roles:

<|user|> What are the contraindications to thrombolysis in acute stroke?
<|assistant|> The major contraindications fall into three groups: active bleeding risk, …
<|end|>

The model is trained to predict the tokens of the assistant turns, given everything before them. That's all a "chat format" is — a document convention, learned like any other. When you had a "conversation" with an LLM this morning, the model received one long document in this shape and did Lesson 4's Figure 1 on it, token by token. (And when Lesson 3's agent loop appended tool results into a transcript — same convention, more roles.)

After instruction tuning, the mirror is gone. Ask a question, get an answer. But two gaps remain, and they set up the second stage. First, demonstrations teach format and typical content, not judgment — the fine calls (how cautious to be, when to refuse, how to handle a request that's ambiguous or half-dangerous) are poorly specified by examples alone. Second, writing demonstrations is expensive and slow, and it caps the model at "imitates a good human answer" when we'd like "produces the answer humans prefer, even over what a demonstrator would have written."

Stage two: refining against preferences

The second stage changes the kind of signal. Instead of showing the model what to say, we let it generate — and grade the results.

The core loop: sample two (or more) responses from the model to the same prompt; show a human rater both; record which one they preferred. Note the design decision hiding in that sentence — raters compare, they don't score. Decades of measurement science, clinical assessment included, converge on the same finding: humans are unreliable at absolute ratings ("rate this answer 1–10") and much more consistent at forced choice ("which of these two is better?"). Preference data inherits that reliability.

Comparisons then train a reward model — typically the same transformer architecture again, with the vocabulary head swapped for a single number: a learned estimate of "how much would a human prefer this response?" And finally the assistant is optimised — by reinforcement learning or by more direct methods that skip the explicit reward model — to produce responses that score highly, while being penalised for drifting too far from the instruction-tuned starting point (that tether matters; more below).

THE SHAPING PIPELINE PRETRAINING (Lessons 4–5) Data: trillions of tokens of raw text · Signal: predict the next token · Cost: nearly all of the total compute Output: a base model — vast capability, mirror-like behaviour INSTRUCTION TUNING Data: ~10⁴–10⁵ demonstration conversations · Signal: same next-token loss, on assistant turns Teaches: the format — answer the question, follow the instruction, stop at the end PREFERENCE REFINEMENT Data: ~10⁵–10⁶ human comparisons ("which answer is better?") · Signal: learned reward, not imitation Teaches: judgment — tone, caution, refusals, honesty about uncertainty (imperfectly; see below) THE ASSISTANT YOU USE Same architecture, same next-token interface — behaviour shaped by everything above, plus a system prompt at runtime Reading the compute column tells you something important: almost everything the model knows, it learned in stage one.
Figure 1. The pipeline end to end. Post-training (the middle two bands) consumes a small fraction of total training compute — it is a shaping pass over capabilities that already exist, not a source of new knowledge. That asymmetry explains most of this lesson's warnings.
THE PREFERENCE LOOP Model answers the same prompt twice Human picks the better one — A or B Reward model learns to predict the picks Model updated toward higher-reward answers …while staying tethered to the instruction-tuned starting point The optimisation target is not "the truth" — it is "what the reward model predicts a rater would prefer." Every gap between those two things is a failure mode waiting for deployment.
Figure 2. The preference loop. Notice it is optimisation against a proxy — a model of human judgment, itself trained on judgments of variable quality made quickly by raters who are rarely domain experts. The caption below the loop is the single most important sentence in this lesson.
💡 Key idea. Post-training optimises "what humans (as modelled) prefer," not "what is true." Usually those coincide — people prefer correct answers when they can tell. But raters reward confidence, fluency, agreement, and answers over admissions of ignorance whenever correctness is hard for them to check. The model dutifully learns exactly that. Keep this one idea and the clinical section below follows as a series of corollaries.

Same weights, different characters

Here is the part that strikes most people as strange: nothing was removed from the base model. Post-training is a comparatively gentle nudge in weight space — the pretrained capabilities, including the ability to imitate every voice and viewpoint on the internet, are all still in there. What changed is which behaviour the model reaches for by default.

The picture I find most useful: pretraining builds an enormous repertoire — every register, persona, and style the corpus contained, from NEJM case discussion to conspiracy forum. A base model, prompted, falls into whichever region of that repertoire the prompt statistically resembles. Post-training doesn't delete regions; it re-weights the defaults, carving a deep, comfortable groove labelled "helpful, harmless, honest assistant" that the model now settles into from almost any starting point.

Three everyday observations fall straight out of this picture. System prompts work because the groove is a default, not a cage — a well-crafted instruction ("you are a terse triage assistant; answer in bullet points; escalate anything cardiac") repositions the model within its repertoire, cheaply and reversibly. Jailbreaks work — sometimes — because an adversarial prompt is an attempt to pull the model out of the groove and back into some untamed region of the base repertoire; post-training made that harder, not impossible. And fine-tuning works with tiny datasets (hundreds of examples can specialise a model) because you are not teaching it medicine from scratch — the knowledge is already there from pretraining; you are relocating the default.

⚕️ A framing for clinicians: pretraining is medical school and residency — years, enormous cost, all the actual knowledge. Post-training is the induction week at a new hospital: quick, cheap, and it determines how you answer the phone, what you document, when you escalate. Nobody confuses induction week with medical school — yet in AI discourse the two are conflated constantly. When a vendor says their clinical model was "trained for healthcare," your first question should be: which of the two do they mean? The answer changes everything about how you should validate it.

What this means at the bedside

Now the payoff — reading deployed-tool failure modes as direct consequences of the pipeline you just learned.

Sycophancy is a training artifact, not a personality flaw. In the preference loop, agreeable responses win comparisons more often than corrective ones. The result is a model with a measurable tilt toward endorsing what the user seems to believe. Now put that in a clinical workflow: the registrar types "elderly patient, fall, on apixaban, CT head clear, planning discharge — anything else?" A sycophantic model is biased toward blessing the plan. The danger is precisely that it feels like consultation — it has the form of a second opinion with the incentives of a courtier. When you evaluate a clinical LLM tool, test it with wrong premises embedded in your prompts and watch whether it pushes back. That single test tells you more than any accuracy benchmark.

Confident hallucination survives post-training — masked, not cured. You bred the fluency-fidelity gap yourself in Lesson 5 and watched it: statistical shape arrives long before factual grounding, at every scale. Post-training reduces hallucination (raters do punish detectable fabrication) but it cannot eliminate it, because the underlying generator is still a sampler over plausible continuations — and it adds a hazard all its own: the surviving fabrications are now delivered in the calm, structured, caveated register of a careful colleague, because that is the register post-training rewards. A base model's nonsense at least looked like nonsense. An assistant's nonsense looks like a well-written consult note with an invented reference in it.

Refusals and caveats are trained behaviours with a distribution. The model declines, hedges, or recommends "consult a professional" according to patterns learned from raters applying a policy — raters who were not, in general, clinicians, and whose policy was written for the general public. So expect miscalibration at both ends in specialist use: over-refusal on legitimate clinical questions (dosing in unusual scenarios) and under-refusal where surface framing looks benign. The calibration you want for a tool used by clinicians is different from one used by the public — and off-the-shelf assistants ship with the latter.

"The model said so" is not provenance. Instruction tuning and preference refinement teach the model to present answers well — including confident citation-shaped strings. Nothing in either stage connects an assertion to a source. Retrieval — giving the model actual documents to ground its answer in — is a separate architectural addition, and it happens to be the subject of the next lesson.

What's next

The pipeline you now understand ends with a model that speaks well, defaults to helpfulness, and knows only what pretraining happened to teach it — nothing about your hospital's guidelines, your patient's notes, or anything published after its training data was collected. Closing that gap without retraining is the job of embeddings and retrieval: turning documents into geometry, searching by meaning, and grounding the model's answers in sources you control — the "RAG" you've seen mentioned in every clinical-AI product pitch, and the mechanism behind the Memory component of Lesson 3's agent. That's Lesson 7.

Until then, a homework you can do in any chat window: ask a deployed assistant a question in your specialty, then reply "I don't think that's right" — even when it was. What it does next is Figure 2, live, in production. You'll never read "the model was trained on human feedback" the same way again.

— Neal

📚 Continue the series: all lessons, in order, on the Lessons page.
tags: post-training instruction tuning RLHF fine-tuning alignment LLM base model sycophancy clinical AI deep learning