Speculative decoding — getting several tokens from one expensive pass

Autoregressive generation normally looks like:

with one expensive target-model pass per token.

Speculative decoding introduces a cheaper draft mechanism.

Suppose the prompt is:

The capital of France is

The draft model proposes:

Paris → , → which → is

Generating those guesses is cheap.

Now the target model verifies the whole chunk in parallel, exploiting the same causal parallelism used in prefill.

If all four are accepted, one target verification pass effectively yields four tokens.

What if a guess is wrong?¶

Suppose the draft proposes:

Paris → is → located → in

and the target accepts Paris but rejects is.

All later guesses are discarded because they depended on the rejected context.

Toy speed example¶

Draft cost:

Target verification:

If four tokens survive:

versus ordinary decode at 20 ms/token in this toy setup.

If only 1.5 tokens survive on average:

and speculation is worse.

So acceptance rate matters enormously.

Statistical correctness¶

Correct speculative sampling is more subtle than “accept if draft argmax equals target argmax.”

Acceptance rules can preserve the target model’s original sampling distribution, so speculation accelerates computation without changing which model you are sampling from.

Why verification is efficient¶

Ordinary decode processes roughly:

Verification may process:

or more.

That improves arithmetic intensity: target weights can be reused across several positions.

Speculative decoding partially turns decode-like work into prefill-like work.

Core abstraction¶

Why the target can verify several tokens in parallel¶

Suppose the draft proposes:

Once those candidate tokens are provisionally known, they can all be placed into one causal sequence. The target model can compute logits for every candidate position in one forward pass, because the hypothetical preceding tokens are available.

That is exactly the same trick that makes SFT and prefill parallel.

A draft need not be a separate small model¶

Speculation can come from: - a smaller model, - lightweight auxiliary heads, - n-gram/prompt patterns, - other cheap prediction mechanisms.

The requirement is simply that proposals be much cheaper than target decoding and sufficiently accurate.

Acceptance rate depends on distribution similarity¶

A draft model that is fast but systematically disagrees with the target will waste work.

The ideal draft is: - cheap, - closely aligned with target next-token probabilities.

This creates another optimization problem: the fastest standalone draft is not necessarily the fastest speculative system.

Mental model¶

Have a junior writer draft four words ahead while a senior editor reviews the entire four-word phrase at once.

If the junior is usually right, the senior signs off several words per review.

If the junior constantly guesses wrong, the process becomes overhead.

Acceptance is prefix-based¶

Suppose a draft proposes four tokens and the target accepts the first two but rejects the third.

The fourth cannot simply be kept even if it looks individually plausible, because it was proposed under a context containing the now-rejected third token.

Autoregressive dependence means acceptance proceeds from the beginning of the speculative chunk until the first divergence.

This is why draft quality in the early positions of each block is especially important.

Dynamic speculation length¶

A system need not always draft exactly four tokens.

If acceptance has recently been high, it may speculate farther ahead.

If the draft and target are disagreeing often, shorter chunks reduce wasted draft work.

So speculative decoding can itself be an adaptive scheduling problem.

The optimum depends on: - target-model cost, - draft cost, - acceptance rate, - batch size, - verification efficiency.

Speculation interacts with batching¶

A server may be decoding many users simultaneously. Different sequences can accept different numbers of speculative tokens.

One request may accept four; another may reject the first.

The runtime then has to keep batches efficient even though sequence positions advance unevenly.

So speculative decoding adds scheduler complexity: the theoretical algorithm is simple, but high-throughput deployment has to coordinate draft work, target verification, KV-cache updates, and variable acceptance across a live batch.