LearnInference & Production ScaleSpeculative Decoding

🚀HardInference Optimization

Speculative Decoding

Reduce LLM inter-token latency by pairing cheap drafting with target-model verification. Learn the rejection-sampling proof, speedup model, method choices, and production rollout gates.

33 min read

Learning path

Step 156 of 196 in the full curriculum

SLM Specialization & Edge Deployment Long Context Window Management

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Consider an on-call alert: inter-token latency has crossed its service-level objective (SLO) while time to first token (TTFT) still looks healthy. Prompt prefill is finished, so the problem sits between streamed tokens. On a low-batch route, the large target rereads weights and KV state for each next-token step. Product wants faster streaming without changing target distribution. Which work can run ahead safely?

The previous chapter treated a small language model as the product: it had to fit the device and do the job itself. Here, a small language model or another cheap proposal path gets a narrower job. It proposes tokens; a larger target model remains the authority.

The draft proposes the next few tokens, then the target scores that known span in one pass. Agreement lets several tokens survive. A disagreement stops the draft at its first mismatch, and the target supplies the correction. That is speculative decoding: fewer target decode calls per emitted token, with the target distribution preserved by modified rejection sampling up to hardware numerical effects.^{[1]Reference 1Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}^{[2]Reference 2Accelerating Large Language Model Decoding with Speculative Sampling.https://arxiv.org/abs/2302.01318}

The trade-off is concrete. Speculation targets low-batch LLM inference that is memory-bandwidth bound; draft work and verification overhead still have to fit the route's latency and throughput budget.

Why one decode step waits on memory

Start with the dependency. An LLM can't commit token $t+1$ until token $t$ exists, so serial decode exposes one target call for every output token. Prefill has many known positions to process together; decode has one new position per step. Speculation attacks that decode loop, not first-token work.

The arithmetic of billions of parameters isn't the first suspect. Modern GPUs are fast at matrix multiplication, but low-batch decode often leaves compute units waiting for weights and cache state to arrive from memory. Transformer sampling is often limited by memory bandwidth, so time to emit one token tracks parameter traffic more than leftover FLOPs.^{[3]Reference 3Fast Transformer Decoding: One Write-Head is All You Need.https://arxiv.org/abs/1911.02150}^{[2]Reference 2Accelerating Large Language Model Decoding with Speculative Sampling.https://arxiv.org/abs/2302.01318} Workload and hardware still decide the bottleneck, so profile before choosing an optimization.

Arithmetic intensity

Arithmetic intensity measures work per byte moved from memory. If one weight load lets the GPU verify 100 candidate tokens, intensity is high. If each load verifies one token, compute units spend more time waiting for the next transfer.

In GPU terms, this ratio is the number of FLOPs (floating point operations, a measure of computational performance) performed per byte of data loaded from memory:

$\text{Arithmetic Intensity} = \frac{\text{FLOPs}}{\text{Bytes Transferred}}$

Reading the formula

A high ratio says each byte carries more math; a low ratio points back to memory traffic as the likely limiter.

During autoregressive decoding, generating one token with a model of $P$ parameters in FP16 or BF16 (2 bytes per parameter) requires, in a weight-only first-order model:

Compute: $\approx 2P$ FLOPs (a matrix-vector multiply against the active weights)
Memory: $\approx 2P$ bytes loaded (those same weights)

Divide and you get an arithmetic intensity of ~1 FLOP/byte in that weight-only model. Chen et al. use the same first-order picture: sampling time tracks parameter size divided by memory bandwidth.^{[2]Reference 2Accelerating Large Language Model Decoding with Speculative Sampling.https://arxiv.org/abs/2302.01318} Real decode also pays for KV-cache traffic, activations, kernels, scheduling, and batching, so profile the engine before declaring a bottleneck.

For an illustrative dense Qwen3.6-27B BF16 target, the round $27 \times 10^9 \times 2$ diagnostic is 54 GB of weights.^{[4]Reference 4Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Published checkpoints can be a bit larger because the true parameter count is not a round 27.0B and extra tensors ship with the card. Plug in deployed accelerator bandwidth before comparing serial decode with verification. The example uses 3,350 GB/s, the H100 SXM HBM3 figure, as a labeled hardware input, not a measured request time.^{[5]Reference 5H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/}

weight-streaming-diagnostic.py

params_b = 27
bytes_per_weight = 2
h100_sxm_hbm3_gbs = 3_350  # NVIDIA H100 SXM spec; replace with the deployed device
weights_gb = params_b * bytes_per_weight
weight_stream_ms = weights_gb / h100_sxm_hbm3_gbs * 1_000

print(f"round BF16 weight footprint: {weights_gb} GB")
print(f"weight-only read time at {h100_sxm_hbm3_gbs} GB/s: {weight_stream_ms:.1f} ms")
print("This is a diagnostic lower bound, not measured request latency.")

Output

round BF16 weight footprint: 54 GB
weight-only read time at 3350 GB/s: 16.1 ms
This is a diagnostic lower bound, not measured request latency.

Phase	Simplified expectation	Bottleneck to measure	Intuition
Prefill (many tokens together)	Higher intensity	Often compute or mixed	Large matrix work amortizes weight loads
Low-batch decode (1 token)	~1 FLOP/byte in FP16 weight-only model	Often memory bandwidth	Move active weights to emit one new token

Two charts for a five-token span: serial decode pays five target weight reads at about 1 FLOP/byte, while one speculative verification pass pays one weight read and raises weight-only intensity to about 5 FLOP/byte. — Count the target weight reads first. Serial decode pays one read per token. One verification pass scores the drafted chunk after a single read, which is why the weight-only intensity climbs from about 1 FLOP/byte toward K.

That arithmetic gives a hypothesis, not a verdict. Measure target decode across served batch sizes, concurrency, precisions, and context lengths; weight traffic is often dominant when the target emits one token at a time.^{[2]Reference 2Accelerating Large Language Model Decoding with Speculative Sampling.https://arxiv.org/abs/2302.01318}

Why verification can win

Take a five-token draft. Serial decode asks target to move its weights five times. Verification feeds those five known positions to target together, so one target pass can replace several serial calls when its extra query and KV work stays small. The target still scores every proposed position, even if the sampler later keeps only a prefix.

That use of known proposed tokens is teacher forcing: target scores a candidate sequence in parallel rather than choosing each token one by one. The draft pays cheap sequential work up front. The exchange is favorable only when target weight movement saved by verification exceeds draft, sampler, and cache costs.

The algorithm

One speculative round from prefix The model: the draft proposes serves, with, and cache. q is draft probability and p is target probability. The target accepts serves and with, rejects cache, and samples latency from the residual as the correction. — Trace left to right from the committed prefix "The model". The draft proposes serves, with, and cache. q is the draft probability and p is the target probability. The target keeps the first two, rejects cache, and samples latency from the residual so the emitted span is "The model serves with latency".

Read the figure as one round you can check by hand. Prefix is "The model"; draft is serves with cache. The target accepts first two tokens, rejects cache, and samples latency as correction. If every draft token survives, the same target pass has logits for the next position, so sampler can emit one extra bonus token. A rejection samples from residual and rewinds KV cache.

Diagram showing Draft K tokens, Target scores the chunk, First reject?, and no. — Draft K tokens, Target scores the chunk, First reject?, and no.

Drafting is sequential on the cheap path. Verification is one target forward pass over the prefix plus proposed tokens. The first rejection stops the round because later draft tokens were generated on a prefix that no longer exists.

Accept/reject criterion

At each proposed position, the target and draft must score the same token under the same prefix: committed history plus any earlier accepted draft tokens. That shared condition makes their probabilities comparable.

If the target assigns at least as much probability as the draft, the verifier always accepts. If the target assigns less, verifier accepts with the ratio of target probability to draft probability; otherwise it rejects and samples a correction.

The running round uses these probabilities:

Position	Draft token	Draft $q$	Target $p$	Verdict
1	serves	0.40	0.60	Accept (target likes it more)
2	with	0.35	0.40	Accept (target likes it more)
3	cache	0.60	0.40	Accept with probability $0.40/0.60 = 2/3$

serves and with are guaranteed accepts because target probability is higher. cache is accepted with probability $2/3$ because target probability is lower. This illustrated draw rejects it, so verifier stops and samples a correction from residual. There is no later draft token after cache in this $K=3$ example; with one, the same stop rule would discard it.

acceptance-probability.py

draft_probability = 0.60
target_probability = 0.40
accept_probability = min(1.0, target_probability / draft_probability)

print(f"accept probability for 'cache': {accept_probability:.2f}")
print("Later drafted positions are discarded after a rejection.")

Output

accept probability for 'cache': 0.67
Later drafted positions are discarded after a rejection.

The construction preserves the target distribution exactly in the mathematical sampler. The accepted branch contributes overlap between distributions; the residual sampler contributes missing mass. Together they reconstruct target probability for every token.^{[1]Reference 1Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

Mathematically, for each draft token $t_i$ , both models assign a probability to that token conditioned on the same prefix. We compare the target model's probability $p_{\text{target}}(t_i)$ with the draft model's probability $p_{\text{draft}}(t_i)$ :

$P(\text{accept } t_i) = \min\left(1, \frac{p_{\text{target}}(t_i)}{p_{\text{draft}}(t_i)}\right)$

Reading the formula

Compute the ratio of the big model's probability to the draft model's probability for this token. If the big model likes it more (ratio >= 1), always accept. If the big model likes it less, accept randomly with probability equal to the ratio. The bigger the disagreement, the more likely rejection.

When a token is rejected at position $i$ , we sample a correction token from the residual distribution:

$p_{\text{residual}}(t) = \frac{\max(0, \; p_{\text{target}}(t) - p_{\text{draft}}(t))}{Z}$

where $Z = \sum_t \max(0, \; p_{\text{target}}(t) - p_{\text{draft}}(t))$ is the normalizing constant.

In plain terms

The correction picks from tokens that the target wanted more than the draft predicted. At the rejected cache position, the draft put 60% on cache and 30% on latency. The target wanted 40% and 50%. That extra 20% on latency is the residual pool. Tokens where the draft was already too generous (cache) get zero residual probability.

Three bar charts at the rejected cache position: draft q puts 60% on cache, 30% on latency, and 10% on batch; target p puts 40%, 50%, and 10%; residual max(0, p-q) is entirely on latency at +20%. — At the rejected cache position, compare draft q, target p, and max(0, p-q). Shared mass can be accepted. The correction sampler draws only from the +20% on latency, so after normalization the residual is a point mass on latency.

This is modified rejection sampling. The accepted branch contributes $\min(p_{\text{draft}}(t), p_{\text{target}}(t))$ ; the residual sampler contributes missing mass $\max(0, p_{\text{target}}(t) - p_{\text{draft}}(t))$ . Add them and you recover $p_{\text{target}}(t)$ exactly.^{[1]Reference 1Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

residual-correction.py

tokens = ["cache", "latency", "batch"]
draft = [0.60, 0.30, 0.10]
target = [0.40, 0.50, 0.10]
overlap = [min(p, q) for p, q in zip(target, draft)]
residual_mass = [max(0.0, p - q) for p, q in zip(target, draft)]
normalizer = sum(residual_mass)
residual = [value / normalizer if normalizer else 0.0 for value in residual_mass]
reconstructed = [left + right for left, right in zip(overlap, residual_mass)]

print(dict(zip(tokens, residual)))
print(f"positive correction mass: {normalizer:.2f}")
print(f"overlap plus residual equals target: {reconstructed == target}")

Output

{'cache': 0.0, 'latency': 1.0, 'batch': 0.0}
positive correction mass: 0.20
overlap plus residual equals target: True

Implementation

The loop below mirrors Leviathan's Algorithm 1 with tiny probability tables, so you can run it without a GPU. A production engine still has two model calls per round: the draft proposes $K$ tokens autoregressively, then the target scores the prefix plus those tokens in one forward pass. In Hugging Face-style causal LMs, softmax turns each logits vector into probabilities, and position $j$ predicts token $j+1$ , so the row at prompt_len - 1 scores the first drafted token.

Before reading code, predict its trace: keep serves and with, reject cache, then draw latency from residual. The sampler must use the same post-processed distributions you serve, not a different temperature or top-p configuration. KV-cache rewind, EOS, batching, and logits processors stay out of this toy.^{[1]Reference 1Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}^{[2]Reference 2Accelerating Large Language Model Decoding with Speculative Sampling.https://arxiv.org/abs/2302.01318}

served-distribution-parity.py

def top_k_normalize(probabilities, k):
    kept = sorted(range(len(probabilities)), key=probabilities.__getitem__, reverse=True)[:k]
    total = sum(probabilities[index] for index in kept)
    return [probabilities[index] / total if index in kept else 0.0 for index in range(len(probabilities))]

raw_target = [0.55, 0.30, 0.15]
raw_draft = [0.40, 0.35, 0.25]
served_target = top_k_normalize(raw_target, k=2)
served_draft = top_k_normalize(raw_draft, k=2)
token_id = 1

raw_accept = min(1.0, raw_target[token_id] / raw_draft[token_id])
served_accept = min(1.0, served_target[token_id] / served_draft[token_id])
print(f"raw acceptance: {raw_accept:.3f}")
print(f"served top-k acceptance: {served_accept:.3f}")
print("Verification must use served probabilities.")

Output

raw acceptance: 0.857
served top-k acceptance: 0.756
Verification must use served probabilities.

speculative-round.py

from random import Random

VOCAB = ("cache", "latency", "batch")

def residual_mass(target, draft):
    return [max(0.0, p - q) for p, q in zip(target, draft)]

def normalize(mass):
    total = sum(mass)
    if total == 0.0:
        raise ValueError("residual has no positive mass")
    return [value / total for value in mass]

def sample(probs, rng):
    draw = rng.random()
    cumulative = 0.0
    for token, prob in zip(VOCAB, probs):
        cumulative += prob
        if draw < cumulative:
            return token
    return VOCAB[-1]

def verify_draft(draft_tokens, token_q, token_p, position_q, position_p, rng):
    accepted = []
    for index, token in enumerate(draft_tokens):
        accept_p = min(1.0, token_p[index] / token_q[index])
        if rng.random() < accept_p:
            accepted.append(token)
            continue
        residual = normalize(residual_mass(position_p, position_q))
        correction = sample(residual, rng)
        return accepted + [correction], "reject"
    bonus = sample(position_p, rng)
    return accepted + [bonus], "bonus"

draft_tokens = ["serves", "with", "cache"]
token_q = [0.40, 0.35, 0.60]
token_p = [0.60, 0.40, 0.40]
# Full distributions at the rejected cache position.
position_q = [0.60, 0.30, 0.10]
position_p = [0.40, 0.50, 0.10]
overlap = [min(p, q) for p, q in zip(position_p, position_q)]
reconstructed = [left + right for left, right in zip(overlap, residual_mass(position_p, position_q))]

emitted, outcome = verify_draft(
    draft_tokens, token_q, token_p, position_q, position_p, Random(1)
)

print(f"emitted: {' '.join(emitted)}")
print(f"outcome: {outcome}")
print(f"reconstructed target: {reconstructed == position_p}")
assert emitted == ["serves", "with", "latency"]
assert outcome == "reject"
assert reconstructed == position_p

Output

emitted: serves with latency
outcome: reject
reconstructed target: True

Tracing one step

The prefix is already "The model". The draft proposes serves, with, cache. With seed 1, verifier accepts first two because target assigns each at least as much probability as draft, then rejects cache ( $r = 0.76 > 2/3$ ). Residual at that position is a point mass on latency, so emitted text is "serves with latency". Any later draft tokens would be discarded; $K=3$ has none.

If every drafted token is accepted, the same target pass has logits for the next position, so sampler emits one bonus token and the round returns $K+1$ tokens. In a real engine, rejection also rewinds target KV cache to accepted prefix before appending correction. Leaving rejected keys in cache would make later attention read tokens that are not in sequence.

Speedup analysis

Once exactness is clear, ask how much target work one round replaces. The expected tokens per verification step depends on acceptance rate $\alpha$ and speculation length $K$ . Start with a deliberately simple model: each drafted token is accepted independently with probability $\alpha$ . Verification still runs left to right, so each accepted token extends the prefix and first rejection ends the draft run with a target correction.

Under that approximation, the expected tokens per verification round is given by the geometric series (with the convention $\mathbb{E}=K+1$ when $\alpha=1$ ):

$\mathbb{E}[\text{tokens per step}] = \frac{1 - \alpha^{K+1}}{1 - \alpha}$

Expected tokens per round

$\alpha$ is the per-token acceptance probability, and $K$ is the number of proposals. The sum counts one token for a first rejection or bonus, then one more token for each accepted position. High $\alpha$ moves the result toward $K+1$ tokens per round; low $\alpha$ leaves the result near one.

Wall-clock speedup

Tokens per round are not wall-clock speedup. Include draft cost ratio $c$ (draft time relative to normal target decode step) and verify cost, which grows as $K$ or sequence length grows. The model below is a first prediction, not a production forecast:

$\text{Speedup} \approx \frac{\mathbb{E}[\text{tokens per step}]}{c_{\text{verify}}(K,s) + K \cdot c}$

The classic Leviathan-style simplification sets $c_{\text{verify}} \approx 1$ (one target pass ≈ one decode step) and $c$ as draft cost relative to that step. That's optimistic when:

Verification attends over $K$ query positions on a long prefix, so attention and KV traffic scale with $K$ and sequence length $s$ . Prefer a profiled $c_{\text{verify}}(K,s) \ge 1$ , often closer to $1+\varepsilon K$ than to a flat 1.
Continuous batching shares a token budget across streams. Each accepted draft multiplies tokens scheduled that iteration; high $K$ can steal slots from other users and worsen multi-tenant ITL even when single-stream speedup looks good.
Real systems also pay for cache growth, kernel launches, sampler parity checks, and tree-attention variants (Medusa/EAGLE), which aren't the classic residual chain alone.

For the first table below we still use $c_{\text{verify}}=1$ so the arithmetic stays readable. Treat those numbers as an optimistic upper-bound sketch to compare against a benchmark, not as promised fleet speedup. The second table and lab re-open $c_{\text{verify}}=1+\varepsilon K$ so you can see how verify cost collapses the win as $K$ grows.

Classic serial two-model drafting pays roughly $K \cdot c$ for $K$ sequential draft steps (each step ~ $c$ target-equivalents). Tree methods such as Medusa and EAGLE replace that chain with parallel heads or a coupled drafter, so proposal cost is not automatically $Kc$ ; profile the method you ship.

Worked example

Use explicit model inputs, not measured results. Suppose a candidate draft path costs 10% of a target pass ( $c = 0.1$ ), set $K = 5$ , and plug in acceptance rate $\alpha = 0.8$ .

Expected tokens per round = $(1 - 0.8^6) / (1 - 0.8) = (1 - 0.262) / 0.2 \approx 3.69$ tokens.

Optimistic cost denominator ( $c_{\text{verify}}=1$ ) = $1 + 5 \times 0.1 = 1.5$ target-equivalent passes.

Optimistic speedup = $3.69 / 1.5 \approx 2.46$ x.

The model predicts about 2.5x for those inputs under flat verification. If acceptance changes to 0.6, same $K = 5$ model predicts about 1.6x; at 0.9, about 3.1x. These are model outputs to compare against a benchmark, not promised throughput.

Model input	$\alpha$	$K$	Assume $c$	$c_{\text{verify}}$	Approx. tokens/round	Approx. speedup
$\alpha$ sweep	0.6	5	0.1	1	2.4	1.6x
$\alpha$ sweep	0.7	5	0.1	1	2.9	2.0x
$\alpha$ sweep	0.8	5	0.1	1	3.7	2.5x
$\alpha$ sweep	0.9	5	0.1	1	4.7	3.1x
Depth sweep	0.85	8	0.1	1	5.1	2.8x
Depth sweep	0.90	10	0.1	1	6.9	3.4x

Nothing in this model table is a hardware measurement. A measured speedup needs target and draft checkpoints, engine and kernel versions, accelerator and precision, batch and concurrency, prompt/output-length distribution, decoding settings, warmup policy, and a named non-speculative baseline. Pair p50/p99 TTFT and ITL with throughput, cost, acceptance, and an output-distribution or correctness check.

Now keep $\alpha = 0.8$ and $c = 0.1$ , but set $c_{\text{verify}} = 1 + 0.05K$ so verification cost grows with draft depth:

$K$	Tokens/round	$c_{\text{verify}}+Kc$	Speedup vs flat $c_{\text{verify}}=1$
1	1.80	1.15	1.57x (vs 1.64x)
5	3.69	1.75	2.11x (vs 2.46x)
10	4.57	2.50	1.83x (vs 2.29x)

Even with strong acceptance, larger $K$ can lose after verify cost is counted. Production sweep tools therefore need $c_{\text{verify}}(K,s)$ profiles alongside $\alpha$ and $c$ .

modeled-speculation-speedup.py

def expected_tokens(acceptance: float, depth: int) -> float:
    return sum(acceptance**step for step in range(depth + 1))

def modeled_speedup(
    acceptance: float,
    depth: int,
    draft_cost: float,
    verify_cost: float = 1.0,
) -> float:
    return expected_tokens(acceptance, depth) / (verify_cost + depth * draft_cost)

for acceptance in (0.6, 0.8, 0.9):
    estimate = modeled_speedup(acceptance, depth=5, draft_cost=0.1, verify_cost=1.0)
    print(f"acceptance={acceptance:.1f}: optimistic speedup={estimate:.2f}x")

print("--- growing verify cost, alpha=0.8 ---")
for depth in (1, 5, 10):
    verify = 1.0 + 0.05 * depth
    estimate = modeled_speedup(0.8, depth, draft_cost=0.1, verify_cost=verify)
    print(f"K={depth}: c_verify={verify:.2f}, modeled speedup={estimate:.2f}x")

Output

acceptance=0.6: optimistic speedup=1.59x
acceptance=0.8: optimistic speedup=2.46x
acceptance=0.9: optimistic speedup=3.12x
--- growing verify cost, alpha=0.8 ---
K=1: c_verify=1.05, modeled speedup=1.57x
K=5: c_verify=1.25, modeled speedup=2.11x
K=10: c_verify=1.50, modeled speedup=1.83x

Optimistic speedup model with c_verify = 1 and draft cost c = 0.1: a line chart of K = 1, 3, 5, 8, 10 shows 80% and 90% acceptance still rising then flattening, while 60% acceptance peaks near K = 3 and falls; a bar chart at K = 5 shows speedup climbing from 1.3x at 50% acceptance to 3.1x at 90%. — Under the optimistic c_verify = 1 model, raising K helps only while acceptance stays high. At 60% acceptance the curve already peaks near K = 3. Treat these as sketches to beat with a benchmark, not promised fleet speedup.

Pick $K$ from measured acceptance, draft cost, and serving behavior. Start a sweep with single-digit depths, then let route-specific benchmarks choose the operating point rather than a universal default.

choose-speculation-depth.py

def modeled_speedup(acceptance, depth, draft_cost, verify_cost=1.0):
    expected = sum(acceptance**step for step in range(depth + 1))
    return expected / (verify_cost + depth * draft_cost)

measurements = {"acceptance": 0.72, "draft_cost": 0.12}
candidates = {
    depth: modeled_speedup(
        measurements["acceptance"],
        depth,
        measurements["draft_cost"],
        verify_cost=1.0 + 0.05 * depth,
    )
    for depth in (1, 3, 5, 8)
}
best_depth = max(candidates, key=candidates.get)
print({depth: round(value, 3) for depth, value in candidates.items()})
print(f"model-selected K to benchmark: {best_depth}")

Output

{1: 1.47, 3: 1.73, 5: 1.662, 8: 1.435}
model-selected K to benchmark: 3

With growing $c_{\text{verify}}$ , the same acceptance and draft cost prefer a smaller $K$ than the flat- $c_{\text{verify}}=1$ sketch (which selected $K=5$ ). Always re-rank depths under the profiled verify curve.

Continuous batching token budget

Single-stream speedup can still hurt multi-tenant ITL. Continuous batching keeps two ledgers: target positions scored and tokens committed. With target-verification budget $B$ and $S$ active streams, no speculation schedules about $S$ target positions per iteration. At depth $K$ , target scores about $S K$ proposed positions even when rejection happens early.

Committed output and KV growth follow a different ledger, about $S(1 + \bar{a}K)$ for accepted fraction $\bar{a}$ . Acceptance reduces committed growth; it doesn't erase verification work. If either target or KV-append capacity fills, streams wait or shrink and multi-tenant inter-token latency rises, even when solo speedup looks good.

multi-tenant-token-budget.py

target_token_budget_b = 64
kv_append_budget_b = 64
streams = 16
accepted_fraction = 0.6  # mean fraction of K that survives before reject
for depth in (0, 2, 4, 8):
    target_work = streams * depth if depth else streams
    expected_commits = streams * (1 + accepted_fraction * depth) if depth else streams
    fit = target_work <= target_token_budget_b and expected_commits <= kv_append_budget_b
    print(
        f"K={depth}: target={target_work:.0f}/{target_token_budget_b} "
        f"commits≈{expected_commits:.0f}/{kv_append_budget_b} "
        f"({'fits' if fit else 'OVER budget: multi-tenant ITL risk'})"
    )

Output

K=0: target=16/64 commits≈16/64 (fits)
K=2: target=32/64 commits≈35/64 (fits)
K=4: target=64/64 commits≈54/64 (fits)
K=8: target=128/64 commits≈93/64 (OVER budget: multi-tenant ITL risk)

Draft model choices

Choose draft source by asking two questions: how much does each proposal cost, and how often does target accept it? A weak drafter rejects early; an expensive drafter consumes the latency budget. Common serving choices are:

Approach	Draft source	Main advantage	Main trade-off
Smaller same-family model	Separate assistant model with the same tokenizer	Simple exact speculative-decoding setup	Extra model to load and schedule
Medusa heads^{[6]Reference 6Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.https://arxiv.org/abs/2401.10774}	Extra heads attached to the target model	No separate model at inference time	Needs extra training and tree verification
EAGLE / EAGLE-3^{[7]Reference 7EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.https://arxiv.org/abs/2401.15077}^{[8]Reference 8EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testhttps://arxiv.org/abs/2503.01840}	Target-coupled speculator over hidden states or direct token heads	Strong proposals without a full second model	More integration complexity
MTP heads^{[9]Reference 9Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}	Checkpoint-native multi-token prediction modules	No separate assistant when supported	Requires checkpoint and engine support
Prompt Lookup^{[9]Reference 9Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}	Reuse repeated n-grams from context	No extra model or training	Only helps when the context repeats itself
Suffix decoding^{[9]Reference 9Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}	Reuse matching suffixes from previous outputs	No extra model	Fit depends on reusable prior output patterns

For classic direct-token probabilistic sampling, draft and target need the exact same tokenizer and token-ID mapping. If IDs map to different subwords, target is verifying a different candidate sequence. Some engines expose limited heterogeneous-vocabulary paths, but those can constrain sampling method; decoded strings alone are not a proof of compatibility.^{[9]Reference 9Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}

tokenizer-compatibility-gate.py

draft_vocab = {"return": 14, " label": 88, " expires": 103}
target_vocab = {"return": 14, " label": 88, " expires": 104}
required_pieces = ["return", " label", " expires"]

mismatches = [
    piece for piece in required_pieces
    if draft_vocab.get(piece) != target_vocab.get(piece)
]
print(f"token-id mismatches: {mismatches}")
print(f"direct draft path allowed: {not mismatches}")

Output

token-id mismatches: [' expires']
direct draft path allowed: False

Medusa: multi-head speculative decoding

Medusa avoids a separate draft model. It adds extra prediction heads to the target itself. Each head predicts a different future position from the same hidden state, so the draft is a tree of continuations rather than one chain. Tree attention then scores those candidate paths in one target pass and keeps the longest accepted prefix.

That removes the job of loading and scheduling a second model, at the cost of training the heads and implementing tree verification.^{[6]Reference 6Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.https://arxiv.org/abs/2401.10774}

The lossless path still uses the same residual sampler as the rest of this lesson. Medusa also describes typical acceptance, which keeps drafts that look plausible under an entropy threshold instead of reconstructing $p$ exactly. Use typical acceptance only when you're willing to leave the exact target distribution. If you need Leviathan-style matching, keep rejection sampling.^{[6]Reference 6Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.https://arxiv.org/abs/2401.10774}

EAGLE drafts from target-model internals instead of a separate full assistant. Earlier variants predict feature states, which can raise acceptance because those states carry more information than a plain token-only head.^{[7]Reference 7EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.https://arxiv.org/abs/2401.15077} EAGLE-3 switches to direct token prediction, fuses low, middle, and high target layers, and trains the drafter with a training-time-test loop on its own outputs. Its paper reports up to 6.5x over vanilla autoregressive generation across five tasks, plus roughly 1.4x latency improvement over EAGLE-2 at batch size 1. In an SGLang test on H100 with LLaMA-Instruct 3.1 8B and MT-Bench, using chain length 3 without the tree structure, EAGLE-3 reports 1.38x throughput at batch size 64. Those are study-specific baselines, hardware, workloads, and settings, not portable service guarantees.^{[8]Reference 8EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testhttps://arxiv.org/abs/2503.01840} Current vLLM docs expose EAGLE-family speculation, but flags and caveats move quickly.^{[9]Reference 9Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}

Prompt lookup decoding

Prompt Lookup Decoding (PLD) skips the neural draft model. It searches the current context window for matching n-grams and reuses them as draft tokens. That helps on tasks with repeated text or pattern matching.

Aspect	How it works
Draft source	Match n-grams from the prompt/context window
Candidate workloads	Code completion, summarization, repetitive text
Memory overhead	No extra model weights
Main failure mode	Little benefit when the context has little repetition

The algorithm scans the context window for n-grams (typically 3-5 tokens) that match the end of the currently generated sequence. When it finds a match, it looks at what token followed that n-gram earlier in the context window and uses that as the next draft token. For example, if the model has generated "timeout error" and the context contains "timeout error repeats Friday," PLD proposes "repeats" as the next draft token.

PLD is a candidate for code generation and other repetitive tasks because variable names, function calls, and boilerplate often reappear within the context window.^{[9]Reference 9Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}

PLD's appeal is that there's no extra model to load, train, or keep in memory. You can stack it with other speculative methods or use it as a fallback when no neural draft model is available.

prompt-lookup-candidate.py

context = "timeout error repeats Friday. auth callback needs review. timeout error"
tokens = context.split()
suffix = ["timeout", "error"]

proposal = None
for index in range(len(tokens) - len(suffix)):
    if tokens[index:index + len(suffix)] == suffix:
        proposal = tokens[index + len(suffix)]
        break

print(f"matched suffix: {' '.join(suffix)}")
print(f"lookup proposal: {proposal}")

Output

matched suffix: timeout error
lookup proposal: repeats

Production deployment

The production decision starts with a bottleneck, not a flag. Speculation spends extra FLOPs on drafting to save target-model memory bandwidth. Modern serving stacks expose several proposer families, so payoff depends on method, route, and load.^{[9]Reference 9Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/} On a compute-saturated route, the same trade can lower throughput and raise cost.

When the classic draft-model setup helps (and when it doesn't)

Before rolling out a separate draft model, check whether your workload benefits from the draft-then-verify cycle. The technique is a trade-off: it burns additional compute (FLOPs) to save memory bandwidth. If your system is already compute-bound, this trade-off will usually backfire and reduce overall throughput.

Scenario	Hypothesis before benchmark	Why test it
Single-user, low-batch inference	Strong candidate	Target decode may be memory-bandwidth bound
Throughput-maximized batching	Measure carefully	Extra draft work can compete with saturated compute
Long outputs	Candidate	More decode steps can amortize setup
Very short outputs	Weak candidate	Setup and drafting may dominate
Repetitive outputs (code, templates)	Candidate	Draft or lookup acceptance may be higher
Diverse outputs	Measure carefully	Acceptance may vary with sampling and prompt mix

Four-quadrant fit map for speculation: low QPS memory-bound traffic is a canary candidate; low QPS compute-bound traffic often loses to extra draft FLOPs; high QPS still-latency routes prefer n-gram or suffix first; saturated compute-bound serving should hold speculation. — Read the quadrant before you sweep K. Low-QPS memory-bound decode is the canary. Saturated compute-bound serving should keep a non-speculative fallback. High-QPS routes that still care about latency can start with n-gram or suffix, which add little extra load.

Current vLLM docs describe model-based methods (EAGLE, MTP, draft model, and related proposers) as stronger inter-token-latency options, while n-gram and suffix decoding add less workload during peak QPS (queries per second).^{[9]Reference 9Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/} Speculation targets the gap between emitted tokens, not TTFT; draft setup can add first-token work. Feature incompatibilities and hardware numerics sit outside the paper proof, so validate the method on your stack.

Serving-engine reality

Serving-engine support changes quickly. vLLM's current docs list several speculation families, plus known feature incompatibilities, and they separate theoretical losslessness from what you should expect under real hardware numerics.^{[9]Reference 9Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/} Treat framework support as something you validate in your stack, not as a timeless property of the algorithm.

Make the rollout comparison reproducible before tuning $K$ . Freeze target and draft checkpoints, engine and kernel versions, accelerator, precision, decoding settings, warmup policy, concurrency, and a prompt/output-length distribution that represents route traffic. Measure baseline TTFT, p50/p99 inter-token latency, and tokens-per-second throughput separately.

Then sweep proposer and depth on those same requests. Break out acceptance, output length, fallback rate, cost, and p50/p99 latency by workload class. Keep a non-speculative fallback for peak-QPS periods or incompatible features, and promote only after output-distribution parity or task correctness passes alongside latency and throughput gates.

The gate example below uses synthetic values to show decision logic, not a benchmark result.

speculation-canary-gate.py

baseline = {"p95_itl_ms": 46.0, "throughput_tps": 380, "sampler_parity": True}
canary = {"p95_itl_ms": 29.0, "throughput_tps": 372, "sampler_parity": True}
minimum_throughput_ratio = 0.95

promote = (
    canary["sampler_parity"]
    and canary["p95_itl_ms"] < baseline["p95_itl_ms"]
    and canary["throughput_tps"] >= baseline["throughput_tps"] * minimum_throughput_ratio
)
print(f"inter-token latency improved: {canary['p95_itl_ms'] < baseline['p95_itl_ms']}")
print(f"canary promoted: {promote}")

Output

inter-token latency improved: True
canary promoted: True

When speculation backfires

Speculation fails in recognizable ways. Diagnose first mismatch before changing $K$ .

Symptom	Likely cause	Fix
Speedup is near 1x or negative	Draft model is too slow or too inaccurate (low acceptance rate)	Benchmark a smaller or better-aligned draft, or switch to Prompt Lookup for repetitive tasks
Correctness checks fail	Acceptance test uses different temperature or top-p than the served model	Ensure the verifier and the sampler share the exact same post-processed distribution
Memory usage spikes unexpectedly	KV cache wasn't truncated after a rejected token	Implement cache rewind so rejected draft tokens don't persist in the cache

Replay representative prompt and output-length distributions through baseline and speculative paths with identical sampling settings, warmup, hardware, precision, engine, and concurrency. Check output-distribution parity and KV-cache rewind first. Then compare acceptance, TTFT, p50/p99 inter-token latency, throughput, cost per output token, and fallback rate by workload class. Promote routes that beat the latency target without breaking the throughput floor; keep a non-speculative fallback for rejection-heavy or incompatible traffic.

Next Step

Continue to Long Context Window Management

Speculation amortizes one target weight read across a short drafted span. Long prompts make that same pass more expensive, because the KV cache grows with every accepted token and attention work rises with sequence length. The next chapter covers that growth: KV-cache math, <span data-glossary="prefill">prefill</span>-vs-decode trade-offs, <span data-glossary="rope">RoPE</span> scaling, and when to use long-context inference versus retrieval.

PreviousSLM Specialization & Edge Deployment

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Fast Inference from Transformers via Speculative Decoding.

Leviathan, Y., Kalman, M., & Matias, Y. · 2023 · ICML 2023

https://arxiv.org/abs/2211.17192

Accelerating Large Language Model Decoding with Speculative Sampling.

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper · 2023

https://arxiv.org/abs/2302.01318

Fast Transformer Decoding: One Write-Head is All You Need.

Shazeer, N. · 2019 · arXiv preprint

https://arxiv.org/abs/1911.02150

Qwen3.6-27B

Qwen Team · 2026

https://huggingface.co/Qwen/Qwen3.6-27B

H100 GPU

NVIDIA · 2026

https://www.nvidia.com/en-us/data-center/h100/

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.

Cai, T., et al. · 2024 · ICML 2024

https://arxiv.org/abs/2401.10774

EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.

Li, Y., et al. · 2024 · ICML 2024

https://arxiv.org/abs/2401.15077

EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

Li, Y., Wei, F., Zhang, C., & Zhang, H. · 2025

https://arxiv.org/abs/2503.01840

Speculative Decoding

vLLM Team · 2026 · vLLM Documentation

https://docs.vllm.ai/en/latest/features/speculative_decoding/

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Speculative Decoding

What is speculative decoding trying to reduce: model quality, target-model calls, or output length?

Why one decode step waits on memory

Arithmetic intensity

Reading the formula

Why does speculative decoding help more in low-batch decode than in a large prefill?

Why verification can win

What does teacher forcing mean in the verification step?

The algorithm

Accept/reject criterion

Why does the verifier stop at the first rejected draft token instead of checking later draft tokens?

Reading the formula

In plain terms

A draft token has draft probability 0.60 and target probability 0.40. What is the acceptance probability?

Implementation

Why must the acceptance test use the same temperature, top-p, top-k, and logits processors as serving?

Tracing one step

Why can a speculative round emit K+1 tokens when all K draft tokens are accepted?

Speedup analysis

Expected tokens per round

Wall-clock speedup

Worked example

Continuous batching token budget

If acceptance drops from 0.8 to 0.5, should you usually increase K first?

Draft model choices

What is the core draft-model trade-off?

Medusa: multi-head speculative decoding

Why do Medusa-style heads remove one major deployment burden of classic speculative decoding?

Prompt lookup decoding

Why is Prompt Lookup Decoding often strong for code completion but weak for open-ended creative writing?

Production deployment

When the classic draft-model setup helps (and when it doesn't)

When is classic draft-model speculation most likely to help production serving?

Serving-engine reality

Which metrics should you split by workload class before deciding speculation is working?

When speculation backfires

What target-model state must be restored after a speculative rejection?

Mastery Check

Discussion