Post-training — how a next-token predictor becomes an assistant

Pretraining teaches:

\[ P(x_{t+1}\mid x_{\le t}) \]

over internet-scale text.

That creates a powerful predictor, but not necessarily a cooperative assistant.

Supervised fine-tuning (SFT)¶

Use conversation-shaped examples:

USER:
What is the largest planet?

ASSISTANT:
Jupiter is the largest planet in our solar system.

The algorithm is still next-token cross entropy.

The key change is the data distribution.

Often loss is applied only to assistant tokens, not user tokens.

So SFT teaches the behavioral grammar:

question → answer
summarize → summary
write code → code

A useful analogy: - pretraining = broad education - SFT = job training

Preference training¶

There are often many valid answers. Humans are better at saying which response is better than writing the single ideal continuation.

Training examples can become:

\[ (x,y_{preferred},y_{rejected}) \]

The objective pushes the preferred response upward relative to the rejected one.

RLHF¶

Classic pipeline:

human preferences
  ↓
reward model
  ↓
generate candidates
  ↓
reinforcement learning
  ↓
updated policy

A KL-style constraint often prevents the tuned model from drifting too far from a useful reference model.

Direct preference methods¶

Methods such as DPO optimize directly from preferred/rejected pairs without requiring the full reward-model + RL loop.

Verifiable reasoning post-training¶

Math, code, games, and tool use can provide objective rewards.

Instead of merely asking:

What answer sounds preferable?

we can ask:

Did the answer actually solve the problem?

Important continuity¶

At inference time, the model is still doing next-token generation.

Post-training changes the probability distribution over trajectories, not the basic Transformer mechanism.

SFT is still teacher forcing¶

During SFT, the model sees the correct previous response tokens while predicting the next one.

For target:

Jupiter is the largest planet.

the training loss predicts is given Jupiter, then the given Jupiter is, and so on.

It does not have to live with its own sampled mistakes during that training pass.

RL changes this because trajectories are actually generated by the policy before they are scored.

Behavior versus capability¶

A pretrained model may already be able to produce a useful answer somewhere in its probability distribution but not reliably do so when instructed.

Post-training can shift probability mass toward: - concise helpful responses, - tool use, - refusal behavior, - structured reasoning, - particular formats.

This is why the distinction between “what the model can sometimes do” and “what it normally does” matters.

KL constraints¶

If reward optimization pushes too aggressively, the model can exploit the reward model or lose broad language abilities.

A reference-policy constraint penalizes excessive drift.

Conceptually:

\[ \text{new behavior gain} - \beta \times \text{distance from reference} \]

This preserves the enormous amount of structure acquired during pretraining while allowing behavior to change.

Modern reasoning post-training¶

Verifiable tasks make preference optimization more objective.

For math/code, the reward can depend on whether the result actually works rather than whether a judge merely likes the wording.

That difference becomes central in reasoning-model training.

Why SFT alone can have exposure mismatch¶

During SFT, the model is conditioned on the correct previous target tokens.

At deployment, it is conditioned on its own sampled previous tokens.

If it makes an early mistake, it may enter a context it rarely encountered during supervised training.

RL rollouts expose the model to states generated by its current policy, which is one reason RL can change sequential behavior in ways that differ from pure imitation.

Preference data carries behavioral values¶

A preferred response can differ from a rejected response in: - correctness, - relevance, - style, - safety, - concision, - tool use.

A preference objective does not inherently know why one was preferred. It simply changes relative probabilities.

Therefore preference-data construction is part of specifying the desired assistant behavior.

Post-training is effectively where broad pretrained capability becomes a particular product personality and policy.

Post-training datasets are much smaller but more targeted¶

Pretraining may involve trillions of tokens.

SFT and preference datasets can be orders of magnitude smaller, yet produce large behavioral changes because they are concentrated on high-value interaction patterns.

This demonstrates an important asymmetry:

Learning broad capability requires enormous diverse experience; steering how that capability is expressed can require much less data.

But small post-training datasets also make quality especially important. A narrow systematic bias in the data can have an outsized effect on user-visible behavior.