Credit assignment — how one reward trains 2,000 reasoning tokens

Yesterday we built this loop:

But we skipped over something rather magical.

Suppose an LLM generates 2,000 tokens of reasoning, submits an answer, and a verifier returns:

How does that single 1 tell gradient descent which of the 2,000 decisions were good?

It doesn't.

That's the credit-assignment problem, and understanding it gets us to the heart of reinforcement learning for reasoning models.

Start with the simplest possible RL rule¶

Suppose the model is solving:

and generates:

17 × 24
→ 17 × (20 + 4)
→ 340 + 68
→ 408

The environment checks:

and returns:

The model generated this trajectory token by token:

The probability of the whole trajectory is:

Take logs:

A very simple reinforcement-learning objective says:

So our gradient contains something like:

Notice what happened.

The final reward gets attached to every token decision in the trajectory.

Conceptually:

17 × (20 + 4) → 340 + 68 → 408
 ↑   ↑   ↑         ↑          ↑
 +1  +1  +1        +1         +1

The model becomes slightly more likely to generate all those choices in similar situations.

That's the basic policy-gradient idea.

But there's an obvious problem¶

Consider a much longer solution:

Step 1   useful
Step 2   useful
Step 3   unnecessary
Step 4   useful
Step 5   wrong
Step 6   compensates for Step 5
Step 7   useful
...
Step 73  lucky correction
Step 74  correct answer

Verifier says:

Naïvely, we reinforce everything.

Including:

Step 3
Step 5
Step 6

The reward tells us:

Somehow, this trajectory worked.

It does not tell us:

Token 847 was the brilliant insight and tokens 911–1052 were useless.

This is temporal credit assignment.

The longer the trajectory, the worse the ambiguity becomes.

Now imagine failure¶

The model generates 2,000 tokens and gets:

Perhaps the first 1,950 tokens contained excellent mathematics.

Then at the end it made one arithmetic typo:

What should training learn?

Ideally:

tokens 1–1950 → mostly good
final arithmetic step → bad

But the outcome reward says only:

The entire trajectory looks equally unsuccessful.

This is why yesterday's process rewards are attractive: they provide denser information about where things went right or wrong.

But process rewards are difficult and expensive to obtain reliably.

So RL needs other tricks too.

First trick: use a baseline¶

Suppose the model receives:

Is that good?

It depends.

If this problem is trivial and the model succeeds 99.9% of the time, reward 1 isn't especially informative.

If the model succeeds only 5% of the time, reward 1 is extremely interesting.

So instead of reinforcing according to raw reward:

we reinforce according to advantage:

where \(b\) is some estimate of expected reward.

Suppose:

Successful trajectory:

Failed trajectory:

Now training says:

success → make this trajectory substantially more likely

failure → make this trajectory somewhat less likely

This is much more useful than treating every success identically.

Where does the baseline come from?¶

One classic approach is to train another network—a value model:

that predicts:

Given the reasoning so far, how much reward do I expect eventually?

Suppose midway through a solution:

We've reduced the problem to x² = 49...

the value model might predict:

Another branch:

Let's assume x is imaginary...

might receive:

Now the model has a notion of whether its current trajectory looks promising.

This should sound familiar from our verifier lesson.

The line between:

and:

can become quite blurry.

A concrete four-step example¶

Suppose a model takes four actions:

and succeeds:

Its value estimates along the way were:

Something interesting seems to happen between \(s_2\) and \(s_3\).

Our expectation jumps:

Perhaps action \(a_2\) discovered the key insight.

A richer RL algorithm can use changes in predicted value to provide more localized learning signals.

Instead of:

everything gets +1

we begin getting something more like:

a1 → mildly useful
a2 → very useful
a3 → useful
a4 → expected completion

Not perfect—but much better credit assignment.

Now we hit the exploration problem¶

Suppose the model currently believes:

Approach A: probability 98%
Approach B: probability  2%

Approach A usually fails.

Approach B actually works beautifully.

If we always choose the highest-probability continuation, we'll almost never discover B.

So RL needs exploration.

Sampling provides some naturally:

Occasionally the model tries unlikely paths.

If one receives unexpectedly high reward, gradient descent increases its probability.

For example:

but B succeeds.

Training might move:

Eventually the formerly rare strategy becomes normal behavior.

This is one reason RL can produce capabilities that ordinary imitation learning may struggle to elicit: the model can discover successful behaviors through exploration rather than only copying demonstrations.

But exploration creates instability¶

Suppose we find one bizarre trajectory that happens to receive reward 1.

Naïve optimization might aggressively increase its probability.

Then the model's behavior changes.

That changes which trajectories it samples.

Those new trajectories change the training distribution again.

Unlike ordinary supervised learning, where the dataset is mostly fixed:

RL training is on-policy-ish:

which changes the next update.

You get a feedback loop:

model
 ↓
behavior
 ↓
reward
 ↓
update
 ↓
different model
 ↓
different behavior
 ↓
...

This can be powerful—and unstable.

This is why algorithms such as PPO exist¶

We've mentioned PPO in passing before.

Proximal Policy Optimization is fundamentally trying to say:

Improve behavior using the reward signal, but don't change the policy too violently in one update.

Suppose before an update:

A reward gradient might want:

PPO effectively constrains how aggressively we trust such updates.

One important quantity is the probability ratio:

If that ratio moves too far from 1, PPO clips the incentive to keep pushing.

Conceptually:

old model
   │
   │ small controlled update
   ▼
new model

rather than:

old model ─────────────────→ wildly different model

This is especially important when reward signals are noisy.

Why reasoning RL is unusual¶

In classic RL, imagine a robot:

move arm
 ↓
pick object
 ↓
reward

Actions might occur a few times per second.

For an LLM, every token is an action.

A 5,000-token reasoning trajectory means roughly:

sequential decisions.

And each decision has a huge action space:

possible tokens.

So we're doing reinforcement learning over extremely long trajectories with enormous branching factors.

Fortunately, pretraining gives us an extraordinary starting policy.

Instead of exploring:

arbitrary token sequences, the pretrained model already assigns almost all probability mass to linguistically and conceptually plausible continuations.

That's another way to understand why pretraining is so foundational.

It massively constrains the search space before RL begins.

Now connect this to distillation¶

Suppose RL discovers an excellent trajectory:

problem
 ↓
clever decomposition
 ↓
correct derivation
 ↓
verification
 ↓
answer

Initially this trajectory might have probability:

Exploration discovers it.

Reward reinforces it.

Eventually perhaps:

Then we can collect these successful trajectories and use them for supervised training or distillation.

So a practical improvement loop can look like:

RL is good at discovery.

Supervised learning is often good at imitation and consolidation.

They complement each other.

The deepest point: reward doesn't contain the solution¶

Suppose the verifier says:

That single bit does not tell the model:

Decompose 24 into 20+4.

It merely says:

Whatever you just did worked.

The actual strategy had to come from somewhere:

• pretraining,
• sampling,
• previous training,
• search,
• tools,
• or chance.

RL then shifts probability toward successful discoveries.

This is why a tiny reward signal can nevertheless produce substantial learning.

The reward isn't transmitting the knowledge directly.

It's performing selection over behaviors the model can already generate.

This is closely analogous to evolution:

Today's mental model¶

Imagine teaching someone to escape a maze while you can only stand outside and ring a bell when they reach the exit.

They try:

left
right
left
straight
right
...

Eventually:

DING!

The bell doesn't tell them which turn was brilliant.

That's the credit-assignment problem.

If you can occasionally shout:

Warmer!

or:

That last turn was promising!

you've created a value/process signal and learning becomes much easier.

And if they already possess a map-reading education—our analogue of pretraining—they aren't randomly bumping into walls. They're exploring plausible routes.

So reasoning RL is roughly: