GRPO — learning by comparing several attempts at the same problem

Yesterday we got to the credit-assignment problem. An LLM generates a 2,000-token solution, receives:

and somehow has to learn which of thousands of token choices deserve credit.

We also saw a traditional solution: train a value model estimating how promising each state is.

Today we'll look at a surprisingly simple alternative:

Instead of asking, “Was this answer good in absolute terms?”, generate several answers to the same question and ask, “Which ones were better than their peers?”

This is the central intuition behind Group Relative Policy Optimization (GRPO).

Start with one math problem¶

Suppose the prompt is:

Our current model independently generates four responses:

A: 381
B: 391
C: 391
D: 401

A deterministic verifier checks them:

A → reward 0
B → reward 1
C → reward 1
D → reward 0

So our group of rewards is:

The average reward is:

Now define a simple relative advantage:

giving:

Training now says:

A → decrease probability
B → increase probability
C → increase probability
D → decrease probability

The crucial thing is that we got a useful baseline without training a separate value network.

The other attempts at the same problem supplied the baseline.

Why is that useful?¶

Remember yesterday's problem.

Suppose a model gets reward:

Is that impressive?

For:

What is \(2+2\)?

not particularly.

For:

Prove this difficult theorem.

very much so.

Raw reward doesn't capture that distinction well.

GRPO asks:

How did this answer perform relative to other answers generated for this exact prompt?

Consider an easy problem:

rewards:

[1,1,1,1,1,1,1,1]

Mean:

so relative advantages are approximately:

There's little to learn.

The model already knows this problem.

Now consider:

[0,0,0,0,0,0,1,0]

Mean:

The successful trajectory has advantage:

That's a strong signal:

Something unusual and valuable happened in trajectory 7.

So group-relative rewards automatically focus training on examples near the model's capability frontier.

This connects directly to our lesson on synthetic-data flywheels.

Usually we normalize as well¶

A simplified GRPO-style advantage is often closer to:

where:

and:

Take:

Mean:

Standard deviation is roughly:

So the successful trajectory gets approximately:

Each failure gets approximately:

Conceptually:

rare success   +1.73   ↑↑↑
failure        -0.58   ↓
failure        -0.58   ↓
failure        -0.58   ↓

The rare successful behavior receives a strong push.

Now connect this to token probabilities¶

Suppose the successful trajectory contains:

23 × 17
= 23 × (10 + 7)
= 230 + 161
= 391

The model generated tokens:

For each generated token, we know the probability assigned by the policy:

Because the trajectory had positive advantage:

gradient updates increase the probability of the actions making up that trajectory.

Very schematically:

Successful group member:

Poor group member:

This is still the policy-gradient machinery from yesterday.

GRPO mainly gives us a convenient way to estimate advantage.

Why not simply train on the successful examples with SFT?¶

This is an excellent distinction.

Suppose we keep:

23 × (10 + 7)
= 230 + 161
= 391

and perform supervised fine-tuning.

SFT says:

Make every token in this response more likely.

RL says something subtly different:

This sampled behavior performed better than expected, so increase its probability relative to the current policy.

That allows reward magnitude, relative performance, exploration and policy constraints to enter the update.

More importantly, the training data is continually generated by the current model.

The loop is:

current model
     ↓
sample group
     ↓
evaluate
     ↓
relative advantages
     ↓
update
     ↓
new current model

So the model continuously explores around its own capability frontier.

There's an important connection to test-time compute¶

A few days ago we learned about generating multiple answers at inference:

problem
 ↓
solution A
solution B
solution C
solution D
 ↓
verify
 ↓
choose best

GRPO does almost the same thing during training:

problem
 ↓
solution A
solution B
solution C
solution D
 ↓
verify
 ↓
compare rewards
 ↓
gradient update

So we can view the relationship as:

used at inference:

used during training:

This is another instance of the loop we've been building:

Why GRPO became attractive for reasoning models¶

Traditional PPO-style RL can involve several large objects.

Conceptually you might have:

policy model
reference model
reward model
value/critic model

For a giant LLM, another trainable model can be expensive.

Suppose your policy is:

A value model of similar scale isn't a small bookkeeping detail.

It means more:

• accelerator memory,
• forward passes,
• communication,
• optimizer state,
• training complexity.

GRPO's cleverness is essentially:

We already generated several answers to this question. Use their relative rewards to estimate what's unusually good or bad.

That can eliminate the need for a separately trained critic/value model in this setup.

Not free—the group sampling itself costs substantial inference compute—but operationally attractive.

The Achilles' heel: you need variation¶

Suppose we sample eight answers:

Mean:

There is no relative signal.

The model learned nothing because it never discovered success.

Likewise:

provides little differentiation.

The most informative group looks something like:

Some success.

Some failure.

That's exactly the capability frontier we discussed earlier:

So good RL training requires a curriculum of problems that are neither impossibly hard nor trivially easy.

As the model improves, yesterday's useful problems become too easy.

The curriculum must move.

There's also a deeper credit-assignment limitation¶

Suppose the winning response is 4,000 tokens long.

GRPO tells us:

But which of those 4,000 token decisions caused the success?

We're still largely reinforcing the whole trajectory.

So GRPO helps solve:

It does not magically solve:

Process rewards, step-level verifiers and other techniques can still provide richer credit assignment.

This distinction is important.

One more subtlety: don't let the model change too quickly¶

As with PPO, we generally don't want one lucky trajectory to radically alter the model.

Suppose an old policy assigned:

to a token.

One successful trajectory shouldn't necessarily push the new policy to:

So GRPO-style objectives typically include PPO-like mechanisms that limit excessively large policy updates, often via probability-ratio clipping, plus regularization toward a reference policy.

The goal is:

because the pretrained model contains an enormous amount of useful structure that our tiny reward signal doesn't capture.

Today's mental model¶

Imagine eight students are given the same difficult problem.

Their scores are:

Student A    20
Student B    25
Student C    95
Student D    30
Student E    85
Student F    20
Student G    35
Student H    25

You don't need an elaborate model predicting:

Given Student C's state after line 17, their expected final score is 73.4.

You can simply observe:

C and E did dramatically better than everyone else on the same problem.

Study what they did and reinforce those behaviors.

Next week, the class gets harder problems.

Again, generate several attempts.

Again, identify the unusually successful ones.

Again, shift behavior toward them.

That's the GRPO intuition:

And notice the remarkable journey we've taken. We started this series with a single next-token probability. We now have that same model generating groups of multi-thousand-token trajectories, executing them in environments, receiving rewards, comparing trajectories, and using those comparisons to alter the probabilities of future tokens.

Yet underneath all of it, the thing we're ultimately adjusting is still: