Skip to content
Novandri.

Speculative Decoding: How Draft Models Speed Up Inference

A small model proposes tokens and a large model verifies them in parallel. Read the mechanism, the speed arithmetic, and when it does not pay.

Rachmadwipa Novandri5 min read

A large language model generates one token at a time, and each step requires a full pass through its weights. That sequential dependency makes inference slow even when hardware is powerful. Speculative decoding attacks the waiting, not the model: a smaller model proposes several tokens, and the large model checks them in one pass.

Direct answer: Speculative decoding uses a draft model to propose a sequence of tokens that a target model verifies in parallel, accepting the ones that match its own distribution and falling back when they do not. The paper Fast Inference from Transformers via Speculative Decoding shows that this can speed up generation while preserving the target model's output distribution. The boundary: the gain depends on draft quality and acceptance rate — a poor draft model adds work without reducing latency.

The mechanism

Three steps repeat:

A three-step flow: a small draft model proposes k tokens, the target verifies them in one pass, and matches are accepted until the first mismatch is corrected.
Conceptual: verification is parallel, so the cost per accepted token falls.
  1. Draft. A smaller, faster model generates k candidate tokens autoregressively.
  2. Verify. The target model evaluates all k tokens in a single forward pass, using them as a proposed prefix.
  3. Accept or correct. Tokens that match the target's distribution are accepted. At the first mismatch, the target's own token is used and the rest are discarded.

The key property is that verification is parallel.

A pass that would normally produce one token now checks several, so the amortised cost per accepted token falls.

The paper's result is that the output distribution is preserved, which matters — this is not an approximation that changes answers.

Three conditions make it work:

The draft must be much cheaper. Otherwise proposal time cancels the saving.

The draft must agree often. Acceptance rate is the main driver of speedup.

Verification must batch efficiently. The hardware and serving stack need to evaluate a prefix in one pass without overhead that erases the gain.

A worked example of the speed arithmetic

The following is hypothetical and illustrative.

A target model takes 40 ms per token. A draft model takes 5 ms per token and proposes k = 4 tokens before verification.

Without speculation: 4 tokens take 4 × 40 = 160 ms.

With speculation: drafting takes 4 × 5 = 20 ms; verification of the four-token prefix takes roughly one target pass, about 45 ms (slightly more than a single-token pass). Total ≈ 65 ms if all four are accepted — about 2.5× faster.

Now vary the acceptance rate:

  • If only the first two tokens are accepted on average, the useful work is 2 tokens in 65 ms, or 32.5 ms per token — still faster than 40 ms.
  • If only one token is accepted, the effective cost is 65 ms per token — slower than baseline.

Three lessons follow:

  1. Acceptance rate is the deciding variable, not draft size alone.
  2. There is a break-even point below which speculation hurts. Measure it on your workload.
  3. k has a trade-off. A larger k offers more potential speedup but a lower probability that all tokens are accepted, and more wasted work on rejection.

The arithmetic is simplified — real serving stacks have batching, memory bandwidth effects, and variable pass costs — but the structure is the paper's.

Where it pays

Latency-sensitive interactive applications where time to token matters and prompts are long enough to amortise setup.

High-volume serving where a modest per-request saving becomes a large cost reduction.

Cases with a good draft pair — a smaller model from the same family often agrees with the target often enough.

Where it does not

Tight memory environments. Loading both a target and a draft model consumes memory that may not be available, as the GPU article explains.

Tasks where the draft disagrees. Highly creative or unusual outputs have lower acceptance rates.

Very short outputs. If the answer is five tokens, setup costs dominate.

The counterargument — that faster hardware makes speculation unnecessary — misses the point.

Hardware improves for both paths; the relative saving remains, though its absolute value shrinks.

What the paper does not promise

Not a universal speedup. The result depends on the draft-target pair and the workload.

Not lower cost in every stack. Serving infrastructure must support batched verification. Without it, the theoretical gain is unavailable.

Not a capability change. Speculative decoding preserves the target's distribution. It does not make the model better, which is why it belongs in the serving layer, not the evaluation conversation.

A decision test

  1. Measure baseline latency per token on your hardware.
  2. Choose a draft model that is much cheaper and closely aligned with the target.
  3. Measure acceptance rate on representative prompts.
  4. Compute the break-even with your actual pass costs.
  5. Tune k against acceptance, not in isolation.
  6. Check memory for both models.
  7. Re-measure after any model or serving update.

For the hardware context, see Why GPU Computing Is Becoming a Strategic Resource; for the serving layer, see Serving Frameworks.

References & Further Reading

  • Yani Leviathan, Diego Alvarez, Francesco Nagle, 2022. Fast Inference from Transformers via Speculative Decoding — The paper showing how a draft model proposes tokens that a target model verifies, speeding up inference without changing output distribution.
  • OpenAI, 2026. Models | OpenAI API — The GPT-6 lineup with published per-million-token input and output prices, context windows, reasoning levels and knowledge cutoffs.