Speculative Decoding

SLM deployment showed how a small model can fit on constrained hardware. Speculative decoding uses a small or cheap draft path differently: not as the final model, but as a proposal engine that a larger target model verifies.

Speculative decoding can speed up generation by letting a cheap draft model propose tokens that a larger target model verifies. This chapter explains the target-distribution proof and the measurements needed before claiming a latency improvement.

Imagine your warehouse uses a slow senior routing checker to write daily shipping reports. The checker reviews every sentence one at a time. Now imagine adding a fast draft scanner that proposes several sentences ahead, and the checker reviews the batch in one pass. If the drafts are good, the checker approves them. If not, it fixes the first mistake and continues. This is the core idea behind speculative decoding: use a fast draft process so a target model can preserve its output distribution in theory while reducing latency when the workload and implementation fit.^[1][2]

This can work because the expensive part of low-batch LLM generation is often repeatedly moving model weights and state, not only arithmetic. A small draft model proposes several tokens, and the large target model scores that proposed chunk in one verification pass instead of spending one full decode step per token.

Speculative decoding targets low-batch LLM inference that is memory-bandwidth bound. Verifying a short drafted chunk still costs extra work, and it helps only when accepted-token savings outweigh the draft and verification overhead.

---

The inference bottleneck

To understand why speculative decoding works, it helps to first understand the fundamental limitations of standard autoregressive generation. In a standard setup, an LLM generates text one token at a time, and each new token depends on all the previously generated tokens. This sequential dependency creates a massive bottleneck.

Most engineers instinctively assume that the slow speed of LLM inference is due to the sheer volume of mathematical calculations required by billions of parameters. However, modern GPUs are incredibly fast at performing matrix multiplications. The real bottleneck lies elsewhere: moving data around. The processor is constantly starved for data, waiting for the massive model weights to be fetched from memory before it can perform its rapid calculations.

Arithmetic intensity

Think of a fulfillment line. Arithmetic intensity measures how much work gets done for each delivery of input data. If one data load lets the GPU verify 100 candidate tokens, that's high intensity, and the GPU is keeping busy. But if each data load verifies only 1 token, the compute units are mostly waiting for the next memory transfer.

In GPU terms, this ratio is the number of FLOPs (floating point operations, a measure of computational performance) performed per byte of data loaded from memory:

$\text{Arithmetic Intensity} = \frac{\text{FLOPs}}{\text{Bytes Transferred}}$

Reading the formula

This ratio helps diagnose whether hardware is spending time computing or waiting for data. Under a weight-only FP16 back-of-the-envelope model, single-token decode performs about 1 FLOP per byte of weights moved.^[2]

During autoregressive decoding, generating one token with a model of $P$ parameters in FP16 (16-bit floating-point) requires:

• Compute: $\approx 2P$ FLOPs (one matrix-vector multiplication per layer)
• Memory: $\approx 2P$ bytes loaded (entire model weights in FP16)

This gives an arithmetic intensity of ~1 FLOP/byte in the weight-only model. Real decode also pays for KV-cache traffic, activations, kernels, scheduling, and batching, so profile the actual engine before declaring a bottleneck.^[2]

For an illustrative FP16 70B target, the weight footprint alone is about 140 GB. Insert a measured or documented device bandwidth into the simplified model before comparing serial decode with verification:

Do not assume low-batch decode is compute-bound without measuring it. Weight traffic is often a dominant cost when the target emits one token at a time.^[2]

The key insight

Here's what makes speculative decoding so clever. Imagine a slow quality-control station where every package must be scanned one at a time. The bottleneck isn't checking the barcode (that's fast); it's moving each package through the scanner separately. Now imagine if the station could inspect 5 proposed packages in one batch and reject only the first bad one. The line moves much faster, even though the station does slightly more work per batch.

That's exactly what happens here: because weight movement dominates, verifying a short candidate chunk can be much closer to one target-model pass than to $K$ separate decode passes. We use a cheap draft model to propose a sequence of candidate tokens. The target model then uses teacher forcing, meaning it scores a known candidate sequence in parallel instead of generating those tokens one by one. The weight-loading cost is paid once for the whole drafted chunk, which raises arithmetic intensity and improves throughput when the draft is accurate enough.

---

The algorithm

The figure is small enough to trace by hand. The draft model proposes "order ships from refund." The target accepts the first three tokens, rejects "refund," and samples a correction from the residual target distribution.

Accept/reject criterion

Think of the target model as a warehouse verifier checking a draft pick list. For each proposed token, the verifier asks: "Would I have chosen this token too?" If the target model likes it even more than the draft model did, instant approval. If the target model likes it less, it might still keep it (proportional to how close the preferences are), or reject it and write its own token instead.

Let's walk through a concrete example. Suppose the prefix so far is "The order" and the draft model proposes three tokens:

For token 1, the target probability (0.60) is higher than the draft probability (0.40), so the verifier always accepts. For token 2, the target probability (0.15) is lower than the draft (0.30). The verifier flips a weighted coin: it accepts with probability 0.15 / 0.30 = 0.50. If the coin comes up reject, the verifier stops checking further tokens and samples a correction from the residual distribution. Token 3 only matters if token 2 survived.

This construction preserves the target distribution exactly. The accepted branch contributes the overlap between the two distributions, and the residual sampler contributes the missing mass. Added together, they reconstruct the target probability for every token.^[1]

Mathematically, for each draft token $t_i$, both models assign a probability to that token conditioned on the same prefix. We compare the target model's probability $p_{\text{target}}(t_i)$ with the draft model's probability $p_{\text{draft}}(t_i)$:

$P(\text{accept } t_i) = \min\left(1, \frac{p_{\text{target}}(t_i)}{p_{\text{draft}}(t_i)}\right)$

Reading the formula

Compute the ratio of the big model's probability to the draft model's probability for this token. If the big model likes it more (ratio >= 1), always accept. If the big model likes it less, accept randomly with probability equal to the ratio. The bigger the disagreement, the more likely rejection.

When a token is rejected at position $i$, we sample a correction token from the residual distribution:

$p_{\text{residual}}(t) = \frac{\max(0, \; p_{\text{target}}(t) - p_{\text{draft}}(t))}{Z}$

where $Z = \sum_t \max(0, \; p_{\text{target}}(t) - p_{\text{draft}}(t))$ is the normalizing constant.

In plain terms

The correction picks from tokens that the big model wanted more than the draft model predicted. If the draft said "the" had 20% probability but the big model wanted 35%, that extra 15% enters the residual pool. Tokens where the draft was already too generous (draft > target) get zero residual probability, since they were over-represented, not under-represented.

This accept/reject scheme is a modified rejection-sampling algorithm. The accepted branch contributes $\min(p_{\text{draft}}(t), p_{\text{target}}(t))$, and the residual sampler contributes the missing mass $\max(0, p_{\text{target}}(t) - p_{\text{draft}}(t))$. Add those two terms together and you recover $p_{\text{target}}(t)$ exactly.^[1]

Implementation

The following code demonstrates a simplified version of the speculative decoding algorithm using PyTorch. It requires two components: a pre-trained target model and a smaller, computationally efficient draft model. The algorithm generates $K$ draft tokens autoregressively with the small model, concatenates them with the current context, and then validates the entire sequence in a single forward pass through the target model.

This version is intentionally pedagogical: it handles the accept/reject loop and residual sampling, but leaves out production concerns like KV-cache reuse, EOS handling, batching, and logits processors. It assumes Hugging Face-style causal-LM logits, where position $j$ predicts token $j + 1$. In a production sampler, the acceptance test has to use the same post-processed distributions you actually serve, not a different temperature or top-p configuration.^[1][2]

Tracing one step

Suppose input_ids currently contains the tokens for "Track order 42". The loop sets step_k = 5 and the draft model autoregressively generates ["ships", "from", "Seattle", "on", "Friday"]. The target model then scores all five draft positions in a single forward pass on the concatenated sequence.

If the verifier accepts the first three tokens but rejects the fourth, the code appends ["ships", "from", "Seattle"] plus a correction token sampled from the residual distribution. The loop then resumes from the new prefix, generating another batch of five draft tokens. If all five tokens are accepted, the code appends all five and also samples a bonus token from the last target logit, yielding six new tokens for one target pass.

---

Speedup analysis

The expected number of tokens generated per verification step depends on the acceptance rate $\alpha$ and the speculation length $K$. A useful back-of-the-envelope model assumes each drafted token is accepted independently with probability $\alpha$ and that one verification pass costs about one normal target-model decode step. Think of this like drafting a pick list where each accepted item keeps the line moving, but the first wrong item forces a stop and correction.

Under that approximation, the expected tokens per verification round is given by the geometric series:

$\mathbb{E}[\text{tokens per step}] = \frac{1 - \alpha^{K+1}}{1 - \alpha}$

Expected tokens per round

$\alpha$ is the per-token acceptance probability and $K$ is how many tokens the draft model proposes. When $\alpha$ is high, you get close to $K+1$ tokens per round because many drafted tokens survive and you often collect the bonus token too. When $\alpha$ is low, most drafts get rejected early and you fall back toward 1 token per round.

Wall-clock speedup

Wall-clock speedup must also account for the cost ratio $c$ (the time it takes to run the draft model relative to the target model). This gives a useful approximation, not an exact production forecast, because real systems also pay for cache growth, kernel launches, and batching effects.

$\text{Speedup} = \frac{\mathbb{E}[\text{tokens per step}]}{1 + K \cdot c}$

The numerator is how many tokens you get per verification round. The denominator is the modeled cost: 1 target-model pass plus $K$ draft-model passes, each costing fraction $c$ of a target pass. For example, if the draft model is 10x cheaper ($c = 0.1$) and you speculate $K = 5$ tokens, the denominator is $1 + 5 \times 0.1 = 1.5$.

Worked example

Use illustrative measured inputs. Suppose a candidate draft path costs 10% of a target pass ($c = 0.1$). You set $K = 5$ and observe an acceptance rate of $\alpha = 0.8$ in a benchmark.

Expected tokens per round = $(1 - 0.8^6) / (1 - 0.8) = (1 - 0.262) / 0.2 \approx 3.69$ tokens.

Cost denominator = $1 + 5 \times 0.1 = 1.5$ target-equivalent passes.

Speedup = $3.69 / 1.5 \approx 2.46$x.

The model predicts about 2.5x for those inputs. If acceptance changes to 0.6, the same K = 5 model predicts about 1.6x; at 0.9, it predicts about 3.1x. These are model outputs to compare against a benchmark, not promised throughput.

The useful $K$ depends on measured acceptance, draft cost, and serving behavior. A small sweep can begin with single-digit depths, but select from route-specific benchmarks rather than a universal default.

---

Draft model choices

The draft mechanism determines your acceptance rate and your overall speedup. It requires a balance: if the draft is too weak, it gets rejected constantly. If it's too expensive, it erases the latency gains from verification. In practice, most systems use one of these families:

In the classic direct-token draft setup, require the separate draft model to share the exact same tokenizer as the target model. If token IDs map to different subwords, the target is not verifying the intended candidate sequence; reject that pairing or use a method that explicitly supports different tokenizers.

Medusa: multi-head speculative decoding

Medusa avoids the need for a separate draft model entirely. Instead, it adds extra prediction heads to the target model itself:

Each Medusa head predicts a different future position from the same hidden state. This design fundamentally changes the architecture from generating a single draft sequence to predicting a tree of possible continuations. Since each head can propose multiple candidates, the candidates form a tree structure rather than a single chain. A specialized tree attention mechanism then evaluates all these candidate paths simultaneously in a single forward pass, efficiently discarding the incorrect branches while keeping the longest accepted path. This approach avoids the overhead and complexity of loading and orchestrating a separate draft model altogether.

The method map above shows that shape visually: hidden state fans out into several future-token heads, and the target verifies the resulting candidate tree.

EAGLE drafts from target-model internals instead of using a separate full assistant. Earlier EAGLE variants predict feature states, which can raise acceptance because those states carry more information than a plain token-only head.^[4] EAGLE-3 moves further in that direction: the paper switches to direct token prediction, fuses low, middle, and high target-model layers, and trains the drafter with a training-time-test loop on its own outputs. The paper reports up to 6.5x speedup in its evaluation setup, but exact gains still depend on engine support, batch shape, and prompt mix.^[5] Current serving docs such as vLLM expose EAGLE-family speculation, but the exact options and caveats change quickly.^[7]

Prompt Lookup Decoding

Prompt Lookup Decoding (PLD) takes a completely different approach. Instead of using any neural model for drafting, it searches the current context window for matching n-grams and reuses them as draft tokens. This non-neural method works surprisingly well for tasks with significant repetition or pattern matching.

The algorithm scans the context window for n-grams (typically 3-5 tokens) that match the end of the currently generated sequence. When it finds a match, it looks at what token followed that n-gram earlier in the context window and uses that as the next draft token. For example, if the model has generated "return label" and the context contains "return label expires Friday," PLD proposes "expires" as the next draft token.

PLD is especially effective on code generation and other repetitive tasks because variable names, function calls, and boilerplate often reappear within the context window.^[6]

The key advantage is simplicity: there's no model to load, no training required, and no memory overhead. PLD can be combined with other speculative methods or used as a fallback when no neural draft model is available.

---

Production deployment

Adding speculative decoding to a production system isn't an automatic win. While it looks promising on paper, real-world performance depends heavily on workload characteristics. For the classic two-model draft-and-verify setup, the core trade-off is simple: you spend extra FLOPs on the draft path to save target-model memory bandwidth. Modern serving stacks now expose several speculation families, so the payoff is method-specific rather than a blanket yes-or-no.^[7] Deploy blindly and you might actually decrease overall throughput and increase costs.

When the classic draft-model setup helps (and when it doesn't)

Before rolling out a separate draft model, you need to check if your workload will actually benefit from the draft-then-verify cycle. The technique is a trade-off: it burns additional compute (FLOPs) to save memory bandwidth. If your system is already compute-bound, this trade-off will usually backfire and reduce overall throughput.

Current vLLM docs position separate draft-model speculation as an inter-token-latency optimization for medium-to-low QPS, memory-bound workloads. The same guide separates that setup from other methods such as EAGLE, MTP, n-gram, and suffix speculation, which have different latency-versus-throughput trade-offs.^[7]

Serving-engine reality

Serving-engine support changes quickly. For example, vLLM's current docs list several speculation families, but they also call out known feature incompatibilities and separate theoretical losslessness from what you can expect under real hardware numerics.^[7] Treat framework support as an operational detail you must validate in your own stack, not as a timeless property of the algorithm.

The practical rollout loop is usually straightforward:

1. Measure baseline inter-token latency and throughput separately.
2. Sweep the draft mechanism and speculation depth on real prompts, not toy strings.
3. Track acceptance rate, output length, and p95 latency by workload class.
4. Keep a non-speculative fallback path for peak-QPS periods or incompatible features.

When speculation backfires

Speculative decoding isn't a universal speedup button. Here are three common failure modes, with symptoms and fixes.

---

Mastery check

Evaluation rubric

• Explain why low-batch autoregressive decode is often memory-bandwidth bound, not math-bound.
• Describe the draft-then-verify loop and why teacher-forced verification improves arithmetic intensity.
• Implement accept/reject logic with probability ratios and residual sampling.
• Analyze speedup as a function of acceptance rate, speculation depth $K$, and draft-to-target cost ratio $c$.
• Compare standard draft-model speculation with Medusa, EAGLE, Prompt Lookup, and serving-engine variants.
• Identify when speculation helps production serving and when it can reduce throughput.
• Debug tokenizer mismatch, sampler mismatch, KV-cache rewind, and "lossless" implementation caveats.

Follow-up questions

How does speculative decoding preserve the target distribution?

It uses modified rejection sampling. Accepted draft tokens contribute the overlap between draft and target distributions, while rejected tokens are replaced by samples from the residual target mass. This is exact in theory when verification uses the same post-processed distribution as serving. Real systems can still differ slightly because of finite-precision kernels and engine details.^[1][7]

What determines speedup?

Acceptance probability $\alpha$, speculation depth $K$, and draft-to-target cost ratio $c$. A useful approximation is $(1 - \alpha^{K+1}) / ((1 - \alpha)(1 + cK))$, but production results also depend on batching, cache behavior, kernels, and workload mix.

How does temperature affect performance?

Temperature changes the served distributions and therefore acceptance; its direction and magnitude depend on target, draft, and prompt mix. Measure acceptance under the actual sampler, and make the verifier use that same processed distribution.

How does Medusa differ from separate draft-model speculation?

A separate draft model proposes one chain with another model. Medusa attaches future-token heads to the target model, proposes a tree of candidates from target hidden states, and verifies those candidates with tree attention.^[3]

When would you not use classic draft-model speculation?

Be cautious in high-QPS, batch-heavy, compute-saturated serving unless benchmarks show a win. Extra draft work can erase latency savings. Other speculation methods can still help, so test by method and workload class.^[7]

What is Prompt Lookup Decoding best for?

It drafts by reusing n-gram continuations already present in context. It works best for repetitive workloads such as code, templates, and some summarization because it needs no extra model but depends on repeated context.^[6]

Common pitfalls

• Output quality drifts only on sampled traffic. Cause: the acceptance test used different temperature, top-p, top-k, or logits processors than the served sampler. Fix: build the ratio and residual from the exact post-processed distribution you serve.
• Acceptance collapses almost immediately. Cause: tokenizer, vocabulary, or prompt-format mismatch between draft and target paths. Fix: verify exact token IDs, chat template, and special-token handling before tuning $K$.
• Speedup stays near 1x or goes negative. Cause: the draft is too slow, too inaccurate, or both. Fix: inspect acceptance rate and draft cost before increasing depth; then shrink the drafter, pick a better-aligned method, or disable speculation on that workload.
• Benchmarks look great on one route and bad on another. Cause: speculation depth was tuned on a global average instead of by workload class. Fix: split latency, acceptance, and cost by prompt family, output length, and QPS band.
• Outputs break after the first rejection. Cause: rejected draft tokens were left in the target KV cache. Fix: rewind the target cache to the accepted prefix before appending the correction token.
• "Lossless" is treated as bitwise-identical across every engine. Cause: the proof is distributional, while real systems still differ in kernels, numerics, and feature support. Fix: compare outputs and latency against a non-speculative baseline on real traffic, not only against theory.^[7]

Check your understanding

Try to answer these questions without looking back at the article.

1. Concrete arithmetic. A 70B model in FP16 has about 140 GB of weights. Under the simplified model, a measured bandwidth input makes one target weight read a large part of low-batch decode time. Why can verifying five draft tokens in one target pass save time?

Sketch: If measured target decode is bandwidth-limited, one verification pass can amortize much of the target weight traffic across several accepted positions. The benchmark still has to account for extra draft work, cache traffic, and rejection.

2. Acceptance probability. A draft token has draft probability 0.4 and target probability 0.2. What is the acceptance probability, and what happens if the token is rejected?

Sketch: Acceptance probability = min(1, 0.2 / 0.4) = 0.5. If rejected, sample a correction from the residual distribution $p_{\text{residual}}(t) = \max(0, p_{\text{target}}(t) - p_{\text{draft}}(t)) / Z$. Tokens the target wanted more than the draft receive positive residual mass; tokens the draft already overestimated receive zero.

3. Negative speedup. You deploy a 1B draft with a 70B target and see only 1.1x speedup. Your acceptance rate is 0.5 and $K = 3$. Should you increase $K$ to 10, switch to a larger draft model, or investigate something else?

Sketch: Low acceptance means most drafts get rejected early. Increasing $K$ wastes effort on tokens that won't survive. A larger draft might raise $\alpha$ but costs more per token. First, check for tokenizer mismatch or temperature mismatch between draft and target. If those are correct, try a smaller, better-aligned draft or switch to Prompt Lookup for repetitive workloads.

---

Key takeaways and next steps

1. Why it works: under the usual back-of-the-envelope model, autoregressive decode runs at about 1 FLOP/byte, so memory bandwidth, not raw math throughput, is the bottleneck.
2. Core algorithm: draft $K$ tokens with a cheap model, verify all $K$ in one target model forward pass.
3. Mathematical guarantee: with the same served distributions and correct residual sampling, modified rejection sampling reconstructs the target distribution in theory; engine numerics and integration still require checks.
4. Speedup model: $\text{Speedup} = \frac{1 - \alpha^{K+1}}{(1 - \alpha)(1 + K \cdot c)}$ is a useful approximation, not an exact production forecast.
5. Draft choices: smaller assistant models, Medusa, EAGLE, and Prompt Lookup Decoding all trade simplicity against proposal quality and integration cost.
6. Production: classic draft-model speculative decoding is most attractive for latency-sensitive, memory-bound workloads, while other speculation methods have more method-specific QPS trade-offs.

You've now seen why speculative decoding works, how the acceptance logic preserves the target distribution, and how to estimate speedup from acceptance rate and draft cost. The core insight is simple: memory bandwidth dominates autoregressive decode, so parallel verification amortizes the weight-loading cost across multiple tokens.