Why reasoning RL is so expensive — one GRPO batch, end to end

Yesterday we saw GRPO conceptually:

There's a systems detail hiding inside that innocent-looking diagram:

Unlike ordinary supervised training, reasoning RL repeatedly runs inference inside the training loop.

That makes it surprisingly expensive.

Let's follow one tiny GRPO batch all the way through the GPUs.

Start with supervised fine-tuning¶

Suppose we already possess this training example:

PROMPT:
Calculate 23 × 17.

TARGET:
23 × 17
= 23 × (10 + 7)
= 230 + 161
= 391

SFT can process essentially the whole sequence in parallel:

Because the correct token \(x_{t+1}\) is already known, we don't need to generate it before computing later positions.

With a causal mask, one forward pass calculates:

simultaneously.

Then:

This should remind you of prefill.

SFT is dominated by large, highly parallel matrix operations—exactly the sort of work GPUs are excellent at.

RL begins very differently¶

Suppose our GRPO group size is:

For the prompt:

Calculate \(23\times17\)

we need eight solutions from the current policy.

But generation is autoregressive.

For each solution:

Token 200 cannot be generated until token 199 exists.

So phase one of our training step is actually inference:

                    PROMPT
                      │
       ┌──────────────┼──────────────┐
       ↓              ↓              ↓
   rollout 1      rollout 2      ... rollout 8
   token-by-      token-by-          token-by-
   token          token              token

Suppose each trajectory averages:

We've just generated:

new tokens before doing any learning at all.

And remember our inference lessons: decode is relatively sequential and often memory-bandwidth-heavy.

Phase 1: Rollout generation¶

For each generated token, the model performs roughly:

The KV cache prevents us from recomputing the entire prefix, but we still need a new target-model decode step.

Conceptually:

8 sequences
    ↓
continuous batch
    ↓
load model weights
    ↓
generate next token for each sequence
    ↓
update 8 KV caches
    ↓
repeat ~2,000 times

This is why all the inference techniques we've learned suddenly matter during training:

• continuous batching,
• efficient KV-cache management,
• GQA,
• optimized attention kernels,
• speculative techniques where applicable,
• high-throughput inference engines.

Reasoning RL isn't purely a training workload.

It's a hybrid inference + training workload.

Phase 2: Evaluate the trajectories¶

Suppose our eight answers receive:

Maybe a math verifier checks the answers.

Or code is compiled and tested.

Or another reward model evaluates them.

Now GRPO computes relative advantages.

Mean:

Standard deviation is roughly:

So successful trajectories get approximately:

and failures:

We now know:

trajectory 1  ↓
trajectory 2  ↓
trajectory 3  ↑
trajectory 4  ↓
trajectory 5  ↑
trajectory 6  ↓
trajectory 7  ↓
trajectory 8  ↑

But we still haven't updated the model.

Phase 3: Recompute token probabilities¶

During rollout generation, we sampled every token from some policy:

To train, we need quantities such as:

for all generated tokens.

Here's the nice part: now the entire trajectory is known.

So we no longer have to process it sequentially.

We can take:

PROMPT + COMPLETE GENERATED RESPONSE

and run it through the model like an SFT example.

All 2,000 positions can be processed in parallel using causal attention.

So the workload switches:

ROLLOUT:

decode
decode
decode
decode
...

to:

TRAINING FORWARD:

████████████████████████
all positions in parallel

This is one of the most important systems distinctions in reasoning RL:

even though they're operating on the same token sequences.

Phase 4: Construct the policy loss¶

For a successful trajectory, suppose:

We want its sampled tokens to become more likely.

For a failed trajectory:

We want them less likely.

Very schematically:

Actual GRPO/PPO-style objectives are more careful.

Recall yesterday's probability ratio:

The objective uses clipping so that one batch doesn't radically change the policy.

There may also be a KL penalty keeping the model near a reference policy.

So we may need information from:

current policy
old rollout policy
reference policy

for the same tokens.

More model evaluations.

Phase 5: Backpropagation¶

Now we're finally back in familiar training territory.

We have a scalar loss:

Autograd computes:

For a huge Transformer, that means backward through:

output projection
MLP
attention
normalization
...
all layers

possibly with activation checkpointing, tensor parallelism, pipeline parallelism, etc.

Then the optimizer updates:

Only now has the model actually learned from those eight trajectories.

Compare the pipelines¶

SFT is approximately:

existing training tokens
        ↓
forward
        ↓
backward
        ↓
update

Reasoning RL looks more like:

                CURRENT MODEL
                      │
                      ▼
             AUTOREGRESSIVE DECODE
             16,000 generated tokens
                      │
                      ▼
                  VERIFIERS
                      │
                      ▼
                 ADVANTAGES
                      │
                      ▼
             TRAINING FORWARD
             over generated tokens
                      │
                      ▼
                   BACKWARD
                      │
                      ▼
                    UPDATE
                      │
                      └────────────↺

And the new model must then generate fresh rollouts because its behavior has changed.

That's a lot more machinery.

A toy compute comparison¶

Suppose an SFT example contains:

tokens.

One training iteration processes those 2K tokens with a forward and backward pass.

Now consider GRPO with:

and 2K-token responses.

Before training, we generate:

tokens autoregressively.

Then we perform training forward/backward over roughly those same:

generated tokens.

Potentially we also evaluate a reference model and reward model.

So compared with one 2K SFT example, the amount of model work can easily be many times larger.

And the rollout portion is particularly awkward because it has decode-like hardware utilization.

This explains why improving inference throughput can directly accelerate model training in modern RL pipelines.

Historically, "training infrastructure" and "serving infrastructure" sounded like separate disciplines.

Reasoning RL increasingly merges them.

There's another problem: policy staleness¶

Imagine 10,000 GPUs generating rollouts.

While they're working, another cluster updates the model:

Some rollout workers may still be generating data using:

By the time those trajectories reach the trainer, the current model is:

The data is now off-policy or stale.

Remember:

If the models have diverged substantially, these ratios can become extreme and the training signal becomes less useful or less stable.

So large-scale RL systems face a scheduling problem:

while also:

and:

This is a distributed-systems problem as much as a machine-learning problem.

An interesting consequence: rollout length has two costs¶

Suppose we encourage a reasoning model to think longer.

Going from:

reasoning tokens potentially means:

During rollout:

more sequential decode tokens.

But it also creates:

more tokens to process during the later training forward/backward.

And attention during the training pass gets more expensive with sequence length.

So long reasoning trajectories can be extremely expensive training examples.

This creates pressure to learn:

rather than merely:

A model that solves a problem correctly in 1,000 tokens can be substantially more valuable than one needing 10,000.

The deeper connection to distillation¶

Suppose our GRPO-trained monster model eventually learns excellent reasoning but routinely spends:

tokens solving problems.

We can collect its successful solutions, perhaps filter or shorten them, and train another model with SFT.

Now the pipeline becomes:

So the full economics can be:

Today's mental model¶

Imagine training a chess player.

SFT is:

Here are one million expert games. Study them.

You can feed those games efficiently because every move is already known.

Reasoning RL is:

Play eight complete games yourself. Then I'll tell you which ones you won. Now replay those games in your head, calculate how probable each move was, work out which games were unusually successful, adjust your instincts, and play eight new games with your updated brain.

The playing is inference.

The replaying and updating is training.

And they alternate continuously: