Speculative decoding — using a small model to make a large model faster

Over the last several lessons, we have focused on distributing work: across GPUs, across sequence positions, and across experts.

Today we look at a very different inference optimization.

Instead of dividing one model's work among more hardware, speculative decoding asks:

Can a cheap model guess several future tokens, and can the expensive model verify those guesses all at once?

The surprising answer is yes — and, when implemented correctly, the output distribution can remain exactly the same as ordinary sampling from the large model.

The central idea is:

\[ \boxed{\text{guess cheaply, verify in parallel}} \]

This attacks one of the most stubborn properties of autoregressive inference: decoding normally advances only one token at a time.

Why ordinary decoding is sequential¶

Suppose the model has already generated:

The capital of France is

To generate the next token, the model computes:

\[ p(x_{t+1}\mid x_{\le t}). \]

Perhaps it produces:

Paris

Only after choosing Paris can it compute:

\[ p(x_{t+2}\mid x_{\le t+1}). \]

So decoding looks like:

large model → token 1
large model → token 2
large model → token 3
large model → token 4

Even if the GPU is extremely powerful, those steps depend on one another.

That sequential dependency is why decode latency is so hard to reduce simply by adding FLOPs.

Introduce a draft model¶

Now imagine we also have a much smaller, faster model.

Call the large model the target model and the small model the draft model.

The draft model predicts several tokens ahead:

prompt:
The capital of France is

draft model:
Paris and it is

Suppose it proposes four tokens:

\[ \tilde{x}_{t+1}, \tilde{x}_{t+2}, \tilde{x}_{t+3}, \tilde{x}_{t+4}. \]

The crucial trick is that the target model can evaluate this entire proposed continuation in one forward pass.

Instead of four separate target-model decode steps:

target → token 1
target → token 2
target → token 3
target → token 4

we do something closer to:

draft → propose 4 tokens

target → verify all 4 positions together

Why is that possible?

Because during verification, the candidate tokens are already known.

Verification looks more like prefill than decode¶

Recall the distinction between prefill and decode.

During prefill, the model receives many known tokens at once:

The capital of France is Paris

and can process their positions in parallel, subject to the causal attention mask.

During ordinary decode, the next token is unknown, so we must generate it before moving on.

Speculative decoding temporarily converts several future positions from:

unknown

into:

proposed

That allows the target model to score those proposed positions together.

This is the deep systems insight:

\[ \boxed{\text{speculation converts some sequential decode work into parallel verification work}} \]

A simple greedy example¶

Suppose the draft model proposes:

Paris is a beautiful city

The target model evaluates the same positions and its greedy choices are:

draft:   Paris | is | a | beautiful | city
target:  Paris | is | a | historic  | ...

The first three draft tokens agree with the target:

Paris is a

The fourth disagrees.

So we accept the matching prefix and then use the target model's token at the mismatch:

Paris is a historic

One target-model verification pass has advanced generation by several tokens.

If agreement is high, this can substantially reduce the number of expensive sequential target-model calls.

But sampling makes this more subtle¶

Greedy decoding is easy to reason about because each model chooses one maximum-probability token.

Sampling is different.

Suppose for a proposed token \(x\):

\[ q(x)=0.20 \]

under the draft model, while:

\[ p(x)=0.30 \]

under the target model.

Here \(q\) is the draft distribution and \(p\) is the target distribution.

A standard speculative-sampling scheme accepts the draft token with probability:

\[ a(x)=\min\left(1,\frac{p(x)}{q(x)}\right). \]

In this case:

\[ a(x)=\min\left(1,\frac{0.30}{0.20}\right)=1. \]

So the token is always accepted.

Now suppose instead:

\[ q(x)=0.40,\qquad p(x)=0.10. \]

Then:

\[ a(x)=\frac{0.10}{0.40}=0.25. \]

The draft model is proposing this token much more often than the target model would.

So we accept it only 25% of the time.

If it is rejected, we do not simply sample naively from \(p\). The correction distribution accounts for probability mass already represented by the draft proposal, with a form proportional to:

\[ \max(0,p-q). \]

This correction is what allows the overall algorithm to preserve the target model's sampling distribution.

That is the remarkable part.

The draft model can change how computation is scheduled without changing which distribution we are sampling from.

The draft model does not need to be equally smart¶

The draft model's job is not to replace the target model.

It only needs to predict what the target model is likely to do.

Suppose the target model is expensive but highly predictable on a continuation like:

The quick brown fox jumps over the lazy

A small model may confidently propose:

dog.

The target model then verifies it.

The small model does not need the target model's full reasoning ability to be useful here.

It needs a high acceptance rate.

If the draft proposes \(k\) tokens and most are accepted, one expensive verification step can advance many positions.

Acceptance rate is the key quantity¶

Suppose the draft proposes:

\[ k=6 \]

tokens per speculative step.

If, on average, only one token survives before a rejection, speculation buys little.

But if the target frequently accepts five or six tokens, the expensive model advances much faster.

A simplified throughput intuition is:

\[ \text{speedup} \approx \frac{\text{tokens advanced}} {\text{target verification cost}+\text{draft cost}}. \]

This is not an exact performance equation because GPU utilization, batch size, memory traffic, and verification shape all matter.

But it exposes the central tradeoff.

A draft model that is extremely cheap but inaccurate may have poor acceptance.

A larger draft model may predict the target better but consume too much time itself.

So the best draft model is not necessarily the smallest one.

We want a good balance between:

\[ \boxed{\text{draft cost}} \]

and:

\[ \boxed{\text{draft agreement with the target}} \]

A worked timing example¶

Suppose ordinary target-model decoding costs:

\[ 20\text{ ms/token}. \]

Generating six tokens normally takes:

\[ 6\times20=120\text{ ms}. \]

Now suppose a draft model proposes six tokens in:

\[ 8\text{ ms}, \]

and the target verifies the six-token block in:

\[ 28\text{ ms}. \]

If all six tokens are accepted, total time is roughly:

\[ 8+28=36\text{ ms}. \]

Instead of:

\[ 120\text{ ms}. \]

That would be a large speedup:

\[ \frac{120}{36}\approx3.3\times. \]

But suppose only two tokens are accepted on average.

Then the same 36 ms of work advances only about two useful positions:

\[ 18\text{ ms/token}, \]

barely better than the original 20 ms/token.

The acceptance rate therefore determines whether speculation pays off.

Why verification can be efficient on GPUs¶

This connects to an earlier lesson about GPU arithmetic intensity.

Single-token decode often performs relatively small matrix operations and repeatedly streams model weights from memory.

Verification processes multiple candidate positions together.

That gives matrix multiplications a larger token dimension:

ordinary decode:

[1 token] × weights


speculative verification:

[several tokens] × weights

The second shape can use GPU compute more efficiently.

So speculative decoding can win twice:

fewer sequential target-model invocations,
better hardware utilization inside each verification invocation.

This is another example of a recurring theme in inference:

The same mathematical model can run much faster when we give the hardware a better-shaped workload.

The KV cache still matters¶

Speculative decoding does not eliminate the KV cache.

The accepted tokens need their key/value states committed to the sequence history.

Rejected speculative tokens must not remain as if they had actually been generated.

So the runtime needs careful bookkeeping:

existing KV cache
      ↓
temporary speculative KV states
      ↓
accept prefix
      ↓
commit accepted states
      ↓
discard or overwrite rejected suffix

This is one reason speculative decoding is not merely an algorithmic trick.

The inference engine must support efficient temporary state management.

Batching makes the problem harder¶

For one request, we might speculate six tokens.

But a production server handles many requests simultaneously.

Imagine:

request A accepts 6 tokens
request B accepts 2 tokens
request C accepts 4 tokens
request D accepts 1 token

The requests now advance by different amounts.

That interacts awkwardly with continuous batching.

The scheduler must keep reorganizing work as sequences move forward unevenly.

So a technique that reduces model calls can simultaneously make scheduling more irregular.

Again we see the recurring tension:

\[ \text{algorithmic efficiency} \quad\leftrightarrow\quad \text{systems complexity}. \]

The draft does not have to be a separate model¶

The basic idea is broader than "small model predicts, big model checks."

Other approaches can produce candidate tokens using:

extra prediction heads
an early-exit portion of the target model
a smaller related model
n-gram or retrieval heuristics
tree-shaped candidate generation

The common structure is:

\[ \boxed{\text{produce cheap candidate futures, then verify them with the target}} \]

Some systems even propose multiple possible continuations, forming a tree rather than one linear draft.

The target can verify many candidate positions in parallel, and the runtime keeps whichever path is valid under the chosen decoding rule.

A concrete mental model: autocomplete plus an editor¶

Imagine a writer dictating a document to a very careful editor.

The slow method is:

writer proposes one word
editor checks it
writer proposes one word
editor checks it
...

Now introduce a fast autocomplete system.

It proposes:

"The meeting has been moved to Tuesday afternoon."

The editor reads the whole proposed phrase at once.

If the editor agrees with:

"The meeting has been moved to Tuesday"

but wants:

morning

instead of:

afternoon

then seven useful words may have been approved in one review.

The autocomplete system is not trusted.

The editor remains authoritative.

But the editor spends less time being asked obvious questions one word at a time.

That is speculative decoding:

autocomplete = draft model
editor       = target model
accepted text = verified token prefix

The speedup comes from making the expensive authority verify batches of guesses instead of originating every token sequentially.

The deeper lesson¶

Autoregressive generation seems to impose an unavoidable rule:

\[ x_{t+1} \rightarrow x_{t+2} \rightarrow x_{t+3} \rightarrow\cdots \]

and logically, it does.

We cannot know the true next sampled token until the previous step is resolved.

But speculative decoding reveals an important distinction:

\[ \boxed{\text{logical sequentiality does not imply all computation must be sequential}} \]

We can perform extra cheap work ahead of time.

If that speculation is correct, the expensive model can validate several sequential decisions using parallel computation.

If it is wrong, we discard the invalid speculation while preserving the target model's behavior.

So today's main principle is:

\[ \boxed{ \text{speculative decoding trades cheap wasted computation for fewer expensive sequential decoding steps} \]

This is a powerful systems pattern far beyond LLMs: when the authoritative computation is expensive and sequential, it can be profitable to predict future work cheaply, execute ahead, and later verify what was correct.