Distillation — moving expensive reasoning into cheaper weights

Over the last few lessons, we built a powerful inference-time recipe:

\[ \text{generate many candidates} \rightarrow \text{verify} \rightarrow \text{search} \rightarrow \text{choose the best} \]

That can make a model much stronger on a difficult problem. But there is an obvious economic drawback:

If the system needs 20 attempts and a verifier every time it sees a familiar kind of problem, we keep paying for the same discovery repeatedly.

Distillation is one way to turn that temporary inference work into persistent model behavior.

The core idea is:

\[ \boxed{ \text{expensive teacher/process} \rightarrow \text{high-quality outputs or trajectories} \rightarrow \text{train a cheaper student} } \]

Start with a concrete example¶

Suppose a strong model can solve:

\[ 17\times24 \]

but only reliably when we let it reason for a while.

An expensive teacher trajectory might be:

17 × 24
= 17 × (20 + 4)
= 340 + 68
= 408

We save:

PROMPT:
17 × 24

TARGET:
17 × (20 + 4)
= 340 + 68
= 408

and use ordinary supervised training on many such examples.

The student receives the teacher's discovered behavior as a target:

\[ L= -\sum_t \log P_{\theta}(y_t\mid x,y_{<t}) \]

Now a behavior that may originally have required search can become much more likely in one forward-generation trajectory.

Distillation does not require a smaller model¶

The word “student” makes people imagine:

huge model → tiny model

and that is common.

But a particularly interesting case is self-distillation.

Let model $M$ solve a problem using expensive inference:

M
↓
32 samples
↓
verification
↓
search
↓
excellent trajectory

Call that expensive system $M^*$.

Now train the original-size model on the successful trajectories:

\[ M \rightarrow M' \]

The architecture might be identical.

What changed is the distribution of behavior.

Suppose the useful trajectory initially had probability:

\[ P_M(\tau)=0.02. \]

Search can find that rare 2% behavior.

Training on successful examples may move it to:

\[ 0.02 \rightarrow 0.08 \rightarrow 0.25 \rightarrow 0.60. \]

Eventually something that required dozens of attempts becomes routine.

That is an important way to understand distillation:

\[ \boxed{\text{search converts compute into data; training converts data into weights}} \]

Distilling answers versus distilling reasoning¶

Imagine a teacher solves:

\[ 73\times18 \]

and returns only:

1314

The student can learn the prompt-answer association, but it receives little signal about the reusable procedure.

A richer teacher example might be:

73 × 18
= 73 × (20 − 2)
= 1460 − 146
= 1314

Now the student sees a behavioral pattern that may transfer to other multiplication problems.

So we can distill:

final answers,
intermediate reasoning,
tool-use sequences,
verification habits,
structured plans,
search-derived trajectories.

This does not mean every visible chain of thought is necessarily the model's true internal algorithm. But as training data, useful intermediate steps can teach a policy that produces better future computation.

Classic soft-target distillation¶

There is another form of distillation that predates modern reasoning models.

Suppose the teacher predicts:

Paris       0.94
Lyon        0.025
Marseille   0.015
London      0.005
...

A hard target would simply say:

Paris = 1
everything else = 0

But the teacher's full probability distribution contains additional structure.

The student can be trained to match:

\[ P_{\text{teacher}} \]

using a loss such as:

\[ L_{KD} = D_{KL} \left( P_{\text{teacher}} \parallel P_{\text{student}} \right). \]

The relative probabilities tell the student that Lyon and Marseille are more plausible alternatives than a random unrelated token.

So distillation can transfer distributional knowledge, not just one correct answer.

The economic argument¶

Suppose an expensive reasoning teacher costs:

\[ \$0.20 \]

per solved training problem.

A cheaper distilled student eventually costs:

\[ \$0.002 \]

per production request.

Generating:

\[ 10M \]

teacher examples would cost, in this toy arithmetic:

\[ \$2M. \]

That sounds enormous.

But if the student then serves:

\[ 10B \]

requests, even a tiny per-request saving can dominate that one-time training expense.

This is the same systems theme we have repeatedly encountered:

\[ \boxed{ \text{spend compute once during training} \rightarrow \text{save compute repeatedly during inference} } \]

Synthetic data creates a flywheel¶

Once a model is capable enough to generate useful solutions, a training loop can become:

MODEL
  ↓
GENERATE
  ↓
VERIFY
  ↓
FILTER
  ↓
TRAIN
  ↓
BETTER MODEL
  ↺

The data is no longer exclusively human-written.

The model can manufacture candidate examples, while verifiers decide which are worth learning from.

This is especially powerful in domains with objective feedback:

code → run tests
math → check answer/proof
tools → inspect real result
games → win/loss
simulation → measured outcome

But synthetic training can go wrong¶

If the model repeatedly generates its own data without strong filtering, errors and biases can become self-reinforcing.

Imagine the model's distribution accidentally underrepresents a rare but important strategy.

Generate synthetic data from that model:

\[ D_1. \]

Train a new model on $D_1$.

The rare strategy may become even rarer.

Repeat:

\[ M_0 \rightarrow D_1 \rightarrow M_1 \rightarrow D_2 \rightarrow M_2. \]

The distribution can progressively narrow.

This is sometimes discussed as model collapse or synthetic-data degeneration.

High-quality external data, diversity, and strong verification remain important.

Distilling search is not the same as learning to search¶

Suppose a teacher spent:

\[ 50,000 \]

tokens exploring many dead ends before eventually producing a polished:

\[ 500\text{-token} \]

solution.

If we train only on that final 500-token trajectory, the student may learn:

Here is how successful solutions look once discovered.

It may not learn:

Here is how to navigate a brand-new problem when the right strategy is not obvious.

This distinction matters.

Distillation is excellent at turning repeated, discoverable behavior into habits.

But hard novel problems may still benefit from fresh inference-time search.

Today's mental model¶

Imagine a senior engineer spends three hours diagnosing a strange production bug.

They try:

hypothesis A → wrong
hypothesis B → wrong
heap profile
database check
new hypothesis
experiment
root cause

Afterward they write a concise postmortem:

If this symptom appears, first compare retained heap objects, then inspect allocation paths around the cache.

A junior engineer studies thousands of these postmortems.

Months later, the same symptom appears.

The junior recognizes it almost immediately.

The original three hours of search have been compressed into expertise.

That is distillation.

\[ \boxed{ \text{temporary computation} \rightarrow \text{training examples} \rightarrow \text{persistent behavior in weights} } \]

And this leads to the next question: if models can generate training data for themselves, can the loop keep improving on its own? The answer depends critically on where the new information and reliable verification come from.