Does RL *create* reasoning, or discover reasoning the model already had?

Yesterday we ended with a puzzle.

A reasoning model might generate 2,000 tokens, get only:

and improve through RL.

But that reward contains almost no information about how to solve the problem.

So where does the reasoning come from?

A useful way to think about it is:

Pretraining creates a vast repertoire of latent behaviors. RL searches among them, reinforces useful combinations, and makes previously rare behaviors much more likely.

RL can eventually push beyond simple selection, but this framing explains a surprising amount about why reasoning post-training works.

Consider a simple debugging problem¶

Prompt:

This Python service's memory usage grows continuously. How would you debug it?

A pretrained model might have many possible continuations:

A: Restart the service.

B: Check for memory leaks.

C: First reproduce the growth, inspect heap profiles,
   compare retained objects over time, identify which
   allocation paths are accumulating...

D: Increase the container memory limit.

E: This could be caused by several factors...

Suppose their probabilities are:

with the remaining probability spread over many others.

The sophisticated debugging procedure already has nonzero probability:

Why?

Because pretraining exposed the model to enormous amounts of:

• debugging discussions,
• source code,
• incident reports,
• textbooks,
• technical reasoning,
• sequences where people diagnose problems.

The model has learned statistical machinery capable of producing useful debugging procedures.

It just doesn't reliably choose them.

Now add RL¶

Generate 64 trajectories.

If C-like behavior occurs with probability:

then the probability of seeing at least one such trajectory is:

That's:

So even though the good strategy appears only 2% of the time, enough exploration will frequently discover it.

Now suppose an environment can verify that this approach actually identifies the leak.

Reward:

A → 0
B → 0
C → 1
D → 0
...

RL increases the probability of C-like trajectories.

Perhaps:

Eventually what once required dozens of samples becomes the model's normal behavior.

No one explicitly taught the model:

Step 1: heap profile. Step 2: compare retained objects...

The reward merely said:

That worked. Do more things like that.

This is very different from learning from scratch¶

Imagine initializing an LLM with random weights and giving it the same reward.

Its output might be:

fQ @ksl zzz %% 3j...

The probability of randomly discovering a coherent 2,000-token debugging procedure is effectively zero.

The search space is astronomical.

With a vocabulary of roughly:

and a 2,000-token trajectory, the naïve sequence space contains:

possibilities.

RL cannot meaningfully search that.

Pretraining has already collapsed this absurd search space onto a tiny manifold of plausible language and behavior.

Instead of considering:

"banana telescope %% jxq"

the model considers things like:

inspect heap
check references
profile allocations
reproduce issue

So you can think of pretraining as constructing an extraordinarily powerful prior over programs expressed in tokens.

RL searches within that prior.

Why call them programs?¶

Consider this token sequence:

Take the original equation.
Subtract 5 from both sides.
Divide both sides by 3.
Check by substitution.

Those aren't merely words.

When interpreted by the model's subsequent computation, they're instructions that alter what happens next.

The sequence acts somewhat like a program:

STATE
 ↓
"subtract 5"
 ↓
new STATE
 ↓
"divide by 3"
 ↓
new STATE

Recall that every generated token becomes part of the context for subsequent tokens.

So generating:

Let's verify this another way.

literally changes the input to all future Transformer passes.

That token can cause later computation to behave differently.

This is one reason intermediate reasoning can be useful.

The model is effectively using its own output as external scratch state.

A small inference example¶

Suppose the model must calculate:

If it tries to emit the answer immediately:

48 × 25 = ?
           ↓
        logits

all computation must somehow happen within the forward pass leading to those answer tokens.

But if it generates:

25 = 100 / 4

that text is added to the context.

The next forward pass now receives:

48 × 25
25 = 100 / 4

Then it generates:

48 × 100 = 4800

Now that becomes context.

Then:

4800 / 4 = 1200

We've transformed one difficult mapping into several easier mappings.

Each generated intermediate token gives us another Transformer evaluation.

So reasoning tokens provide both:

and:

That's why test-time reasoning can genuinely increase computational capability rather than merely make an explanation longer.

What might pretraining have learned?¶

Internet-scale text contains countless reusable computational motifs:

break problem into parts

work backwards

consider edge cases

write an equation

make a table

try an example

compare alternatives

check the answer

debug systematically

search for contradiction

Pretraining doesn't store these as neat functions:

def work_backwards(problem):
    ...

They're distributed through the network's weights.

But prompts and generated tokens can activate these behaviors.

You might think of the model as containing an enormous fuzzy library of learned cognitive subroutines.

Pretraining builds the library.

Post-training changes which subroutines get called, when, and in what sequence.

RL can also discover combinations¶

Suppose the model separately knows how to:

A: decompose a problem
B: use Python
C: inspect tool output
D: reconsider an assumption
E: verify an answer

Perhaps pretraining rarely contains the exact sequence:

But during RL exploration, the model may generate that combination.

If it succeeds, reinforcement makes the entire behavioral pattern more likely.

So RL isn't merely selecting complete memorized solutions.

It can discover new compositions of existing capabilities.

That's much more powerful.

Can RL create genuinely new capabilities?¶

This is where the simple "RL only selects existing behavior" story becomes too strong.

Suppose a behavior initially has extraordinarily tiny—but nonzero—probability.

Repeated updates can gradually move probability mass toward it.

As the policy changes, it begins visiting states it rarely encountered before.

Those states allow new behaviors to be discovered.

So learning can proceed through a staircase:

The final behavior may be extremely unlikely under the original model.

So it's better to say:

rather than:

The distinction matters.

This explains the importance of curriculum¶

Suppose our model can solve difficulty-5 problems:

But difficulty-20 problems:

Training directly on difficulty 20 produces:

0 0 0 0 0 0 0 0

No useful GRPO signal.

Instead:

difficulty 5
     ↓
learn
     ↓
difficulty 7
     ↓
learn
     ↓
difficulty 10
     ↓
learn
     ↓
difficulty 14
     ↓
...

Each stage moves the model into a region from which the next capability becomes discoverable.

This is analogous to teaching mathematics.

You don't go:

You build intermediate capabilities that make later concepts reachable.

There's an important inference consequence¶

Suppose RL makes the following behavior highly probable:

1. understand problem
2. identify ambiguity
3. form hypothesis
4. use tool
5. inspect result
6. revise
7. verify
8. answer

At inference time, the model may now spontaneously generate this trajectory.

Each step modifies the context.

Each step invokes another forward pass.

Some steps invoke external tools.

So what RL has learned isn't simply a better mapping:

It has learned a policy controlling an inference-time computation:

That's much closer to an agent than a traditional classifier.

Today's mental model: a chef's kitchen¶

Imagine a chef who has spent 20 years cooking.

That's pretraining.

They possess thousands of latent techniques:

reduce
sear
deglaze
emulsify
taste
adjust acidity
rest

Now put them in a cooking competition.

They try combinations.

One unusual sequence:

sear → deglaze → reduce → add acid late

wins.

That's reward.

Next competition they're more likely to use that sequence.

Eventually it becomes instinctive.

That's RL.

But the judge didn't teach them how to deglaze.

The judge only said:

This dish won.

The techniques came largely from the chef's prior education; reinforcement selected and composed them into more successful behavior.

For LLMs:

And at inference time those habits can themselves invoke long chains of additional computation.