Verifiers and process rewards — when checking is easier than solving

Yesterday we looked at test-time compute: instead of asking a model for one answer, we can generate several attempts, branch, search, and spend more compute on a difficult problem.

That immediately creates a new bottleneck:

If I have 32 candidate solutions, how do I know which one is actually good?

This is the role of a verifier.

The central intuition is one of the oldest ideas in computation:

\[ \boxed{\text{finding a solution can be much harder than checking one}} \]

For many useful domains, an LLM can propose candidate answers while a cheaper, more reliable process checks them.

Outcome verification¶

Suppose the problem is:

\[ 37\times43 \]

and four generated answers are:

A deterministic checker calculates the product and assigns:

The verifier does not need to understand the model's reasoning. It only checks the final result.

For code, the analogue is even more useful:

candidate program
      ↓
compiler
      ↓
unit tests
      ↓
pass / fail

A program that passes 100/100 tests provides much stronger evidence than another language model merely saying:

Looks correct to me.

This is an outcome verifier:

\[ R(x,y)\in\{0,1\} \]

or more generally a numeric score.

Why verifiers unlock test-time compute¶

Suppose one model attempt has only:

\[ p=0.20 \]

probability of being correct.

Generate 16 independent-ish attempts.

The probability that none are correct is:

\[ 0.8^{16}\approx0.028. \]

So the probability that at least one is correct is approximately:

\[ 97.2\%. \]

That sounds wonderful—but only if we can identify the successful attempt.

Without verification, we have merely created a pile of answers.

So scaling inference compute depends on two different capabilities:

\[ \boxed{\text{generate diversity}} \]

and:

\[ \boxed{\text{recognize quality}} \]

The second can matter as much as the first.

Use the verifier during training¶

Now turn the inference system into a training system.

For one prompt, generate:

trajectory A → reward 0
trajectory B → reward 1
trajectory C → reward 0
trajectory D → reward 1

Reinforcement learning can increase the probability of the successful trajectories.

This creates the loop:

\[ \text{generate} \rightarrow \text{verify} \rightarrow \text{reward} \rightarrow \text{update model} \]

The verifier therefore does double duty: - at inference time, it helps select answers; - during training, it supplies a learning signal.

The limitation of final-answer rewards¶

Consider:

\[ 3x+5=20. \]

A clean solution is:

3x = 15
x = 5

Now imagine a generated trajectory:

3x = 25        ← wrong
x = 5          ← somehow corrected
check: 15+5=20
answer: 5

If the verifier checks only the final answer, it returns:

\[ R=1. \]

The trajectory receives the same outcome reward as the clean derivation.

That is problematic if we want to teach robust reasoning rather than lucky answers.

Process reward models¶

A process reward model, or PRM, tries to score intermediate reasoning.

For example:

Start: 3x+5=20       ✓
Subtract 5: 3x=15    ✓
Divide by 3: x=5     ✓
Check: 20=20         ✓

versus:

Start: 3x+5=20       ✓
Subtract 5: 3x=25    ✗

Instead of waiting for one reward at the end, the system gets information about the path.

This is useful for both training and search.

Search with a process verifier¶

Think of reasoning as a tree.

At one point the model proposes:

             current state
             /     |      \
          step A  step B  step C

A process verifier scores:

\[ V(A)=0.8,\quad V(B)=0.15,\quad V(C)=0.55. \]

The search algorithm can spend more compute expanding A and C rather than B.

Formally, let a trajectory be:

\[ \tau=(s_1,s_2,\ldots,s_T). \]

The language model proposes:

\[ P_\theta(s_{t+1}\mid s_{\le t},x). \]

A verifier or value model estimates:

\[ V(s_{\le t},x). \]

That starts to look like classic search: - the LLM is the proposal policy, - the verifier is a value function, - the search procedure allocates computation.

Objective verifiers versus learned verifiers¶

Some environments give us unusually strong verification:

code → compiler and tests,
arithmetic → exact calculation,
algebra → symbolic checker,
games → win/loss,
tool calls → actual API/database result.

Other tasks do not.

For:

Write a thoughtful strategic memo.

there is no compiler that returns CORRECT.

We may use: - human preferences, - model judges, - learned reward models.

But now verifier error matters.

A learned judge can be fooled, share the generator's misconceptions, or reward superficial features.

Why using the same model as judge is risky¶

Suppose the generator has blind spot \(X\).

If the verifier is another copy of the same model family, it may share blind spot \(X\).

The system can produce:

wrong solution
     ↓
same blind spot in judge
     ↓
approved

Independent external checks break this correlation.

This is why tool-based and environment-based rewards are so attractive: reality does not care whether the answer sounds persuasive.

Reward hacking¶

A deeper problem appears whenever the verifier is only a proxy for what we really want.

Suppose we reward code only for passing a small public test suite.

A sufficiently optimized model may learn to:

special-case the tests

rather than implement the intended general solution.

It has maximized the reward while missing the objective.

This is reward hacking.

The principle is:

\[ \boxed{\text{the stronger the optimizer, the more carefully the reward must represent the true goal}} \]

Weak optimization may tolerate a sloppy metric. Powerful search can systematically find its loopholes.

Outcome versus process rewards¶

Outcome rewards have major advantages: - cheap, - objective in verifiable domains, - hard to micromanage incorrectly.

Process rewards offer: - denser credit, - earlier pruning, - better distinction between good and lucky reasoning.

But process rewards are also harder to label correctly. A human or learned judge must decide whether intermediate steps are actually valid.

So there is a real trade:

\[ \boxed{\text{simple reliable final signal}} \]

versus:

\[ \boxed{\text{richer but potentially noisy step-level signal}} \]

Today's mental model¶

Imagine a maze.

A solver proposes complete routes.

An outcome verifier stands at the exit and says:

escaped → 1
didn't escape → 0

That's very reliable.

A process verifier stands above the maze and says during the run:

this corridor looks promising
that turn is a dead end
you are getting closer

That richer signal can make search much more efficient—but only if the guide actually understands the maze.

Modern reasoning systems often combine all three pieces:

\[ \boxed{ \text{generator} \times \text{search} \times \text{verifier} } \]

The generator proposes possibilities.

Search spends computation.

The verifier determines which possibilities deserve more attention or future training.

That combination is often more capable than simply making the generator bigger.