The synthetic-data flywheel — can models create their own training data?

Last time we saw distillation:

That suggests an almost irresistible idea:

Once a model becomes sufficiently capable, why not have it generate its own training data indefinitely?

Something like:

Could this become a self-improvement loop?

Sometimes, yes. But there's an important constraint:

That's why environments with verifiable outcomes are so important.

Imagine training a coding model¶

Start with model \(M_0\).

Give it 1 million programming problems and ask it to generate four solutions per problem:

Now run each candidate against unit tests.

For one problem:

candidate A → 6/10 tests
candidate B → 10/10 tests  ✓
candidate C → compile error
candidate D → 8/10 tests

Keep B.

Do this across the dataset and perhaps obtain:

verified successful solutions.

Now train \(M_0\) on those solutions:

Suppose \(M_1\) is better.

Now repeat.

Because \(M_1\) solves more difficult problems, round two might produce:

verified solutions.

Train again:

Conceptually:

             ┌─────────────┐
             │    MODEL    │
             └──────┬──────┘
                    ↓
              generate attempts
                    ↓
             ┌─────────────┐
             │ ENVIRONMENT │
             │ tests/tools │
             └──────┬──────┘
                    ↓
                 reward
                    ↓
              train model
                    │
                    └───────────↺

This is a data flywheel.

Notice what's producing the trustworthy information.

It's not primarily the model.

It's the environment.

Why can't the model simply judge itself?¶

Suppose we ask a model:

Write an explanation of quantum gravity.

Then ask:

Is this explanation correct?

It says:

Yes, excellent explanation.

We now train on it.

The problem is obvious.

If the model has misconception \(X\), it may:

1. generate \(X\),
2. fail to recognize \(X\) as a misconception,
3. approve \(X\),
4. train on \(X\),
5. become even more confident in \(X\).

That's a positive-feedback loop in the wrong direction.

Schematically:

This is why yesterday's distinction between generator and verifier matters so much.

Self-improvement becomes much more compelling when verification comes from somewhere the model cannot simply persuade.

Compare three environments¶

Mathematics¶

Model claims:

for:

We substitute:

Excellent verification.

Code¶

Model writes:

def sort(x):
    ...

We run it against:

test cases.

Excellent verification.

Essay writing¶

Model writes:

This is a profound and original interpretation of Hamlet...

What's the objective test?

There often isn't one.

We can use human preferences or learned critics, but the reward signal is inherently less crisp.

So the potential strength of the flywheel differs dramatically across domains.

This leads to a surprisingly important concept: grounding in consequences¶

Consider an AI controlling a simulated robot.

It receives:

GOAL:
put red cube into box

The model generates actions:

move left
grip
lift
move right
release

The simulator reports:

if the cube ends in the box.

Now the model can try:

different action sequences.

The simulator doesn't care whether the model's explanation sounds persuasive.

The cube is either in the box or it isn't.

That's a much stronger training signal than:

Another language model rated this answer 8.7/10.

So environments give models access to consequences.

And consequences generate training information.

This is different from pretraining¶

Remember our original training corpus:

Wikipedia
books
code
web pages
papers
conversations
...

The model learns:

from things humans have already produced.

The information source is fundamentally:

An environment-based training loop is different.

The model can create new experiences:

try action A → fail
try action B → fail
try action C → succeed

Those trajectories may never have existed on the internet.

The model has effectively generated new training information by interacting with something external.

This distinction is profound.

A concrete mathematical example¶

Suppose a model is only:

likely to solve a certain class of problems.

If we ask it once, our training-data yield is poor.

But generate:

independent-ish attempts.

The probability that none succeeds is:

Therefore the probability of finding at least one success is:

So a model that's only 10% reliable can potentially generate high-quality examples for nearly every problem—provided we can perfectly identify the successful attempt.

Then train on those successes.

Maybe the new model becomes:

reliable.

Now fewer samples are required to discover successful trajectories.

That produces the flywheel:

At least in the idealized case.

This is one reason test-time compute, verification and training are so tightly connected.

But where do the problems come from?¶

We've quietly assumed an endless supply of good problems.

That's another bottleneck.

If the model repeatedly trains on:

2+2
3+4
5×6

it quickly saturates.

Improvement requires increasingly difficult tasks near the edge of the model's capability.

This suggests curriculum generation:

model succeeds easily
       ↓
generate harder task

model fails always
       ↓
make task easier

model succeeds sometimes
       ↓
excellent learning opportunity

Ideally, training spends lots of time in the region where:

Too easy:

gives little new learning signal.

Too hard:

produces no successful examples.

The sweet spot lies around the model's capability frontier.

Now things start looking like games¶

This is exactly why games have historically been such powerful AI training environments.

Imagine chess.

We don't need humans to label:

Move Qf3 deserves 8.4 points.

Instead:

AI plays game
      ↓
win / loss / draw
      ↓
reward
      ↓
update policy
      ↓
play stronger games

As the model improves, its opponent can improve too.

Self-play automatically generates an evolving curriculum.

Language reasoning is harder because:

Did this research plan succeed?

is much harder to verify than:

Did White win the chess game?

But wherever we can build reliable environments, similar ideas become possible.

Tool use creates environments¶

This is where agents become relevant.

Give a model access to:

Python
compiler
browser
database
terminal
spreadsheet
simulator

Now instead of merely generating:

I believe the answer is 37.

the model can perform:

hypothesis
    ↓
call tool
    ↓
observe result
    ↓
revise hypothesis
    ↓
call another tool
    ↓
answer

Those tool results provide external information.

And successful tool-use trajectories can become future training examples.

So tools aren't merely useful at inference time.

They can create training environments.

The critical bottleneck moves¶

In early LLM scaling, a central question was:

With synthetic data:

is almost trivial.

A model can generate mountains of text.

The harder questions become:

and especially:

Think back to yesterday.

If we have a perfect verifier, even a mediocre generator can become useful because we can sample repeatedly.

If we have a terrible verifier, an extraordinarily capable generator can poison its own training data.

So the flywheel depends on:

not generator capability alone.

Why this doesn't imply infinite self-improvement¶

Suppose the current model cannot generate a correct solution to some problem at all:

Sampling:

times doesn't necessarily help.

There must be some source of new information.

That might come from:

• human experts,
• stronger teacher models,
• tools,
• search,
• simulations,
• experiments,
• real-world feedback,
• new data.

A closed loop containing only a model repeatedly paraphrasing its existing beliefs has no obvious mechanism for conjuring arbitrary new truths.

This gives us the deepest principle for today:

Today's mental model¶

Imagine teaching yourself physics with a notebook.

You could:

write a theory
read your theory
rate your theory
rewrite your theory

You might improve your prose.

But your misconceptions can survive indefinitely.

Now put yourself in a laboratory:

form hypothesis
      ↓
run experiment
      ↓
observe reality
      ↓
hypothesis wrong
      ↓
revise
      ↓
run another experiment

The laboratory injects new information into the loop.

For AI systems, compilers, unit tests, mathematical checkers, games, simulators, databases, tools and eventually real-world interactions can play the role of that laboratory.

So the interesting self-improvement loop isn't:

It's: