Speculative Decoding

Your large language model starts its answer promptly, but streams out remaining tokens at a frustrating crawl. Prompt processing isn't the culprit. Each next-token decode step stalls waiting for an expensive target forward pass. Could a cheaper model guess several tokens ahead without altering the output distribution users receive?

The previous chapter deployed a small language model as the primary engine answering queries. Here, a cheap mechanism proposes tokens for the target to verify together. It can be a separate model, a trained prediction head, or a lookup in existing text. An exact sampler commits an accepted prefix and corrects the first rejection. This procedure is speculative decoding. The classic algorithm preserves the target's sequence distribution in exact arithmetic, provided its actual conditional probabilities and stopping rules are respected.^{[1]Reference 1Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192[2]Reference 2Accelerating Large Language Model Decoding with Speculative Sampling.https://arxiv.org/abs/2302.01318}

The same seed need not produce the same text: the two algorithms consume random draws differently. Floating-point kernels can also change the probabilities evaluated in serial versus verification passes. Nor is every heuristic called speculative decoding exact. We will prove the sampler identity, exercise a complete probability-level round, and distinguish a performance model from a measured serving gain.

Why memory bandwidth throttles autoregressive decoding

Standard target sampling has a sequential dependency: its next-token distribution conditions on tokens already selected. A transformer can score a known continuation in parallel with causal masking, but the next unknown sample is needed before the following ordinary decoding step.

Batch-one dense decoding is often limited by weight traffic, though attention, cache traffic, communication, and launch overhead can dominate particular workloads. Consider a hypothetical model using $P=30\times10^9$ matrix weights per position in BF16. Assume each weight is fetched once from HBM per pass and multiply-add counts as two FLOPs. That gives 60 GB of weight traffic and about 60 GFLOPs per position. This approximation excludes attention and other work; an embedding lookup does not read every stored embedding row.

Under these assumptions, the weight contribution to arithmetic intensity is:

$I = \frac{2P \text{ FLOPs}}{2P \text{ bytes}} = 1\text{ FLOP/byte}.$

NVIDIA lists 3.35 TB/s and 80 GB for H100 SXM. Its advertised BF16 rate of 1,979 TFLOP/s includes structured sparsity; the corresponding dense rate is about 989.5 TFLOP/s.^{[3]Reference 3H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/} The weight-only peak-rate floors are:

$T_{\text{memory}} = \frac{60 \times 10^9\text{ bytes}}{3.35 \times 10^{12}\text{ bytes/second}} \approx 17.91\text{ ms},$

$T_{\text{compute}} = \frac{60 \times 10^9\text{ FLOPs}}{989.5 \times 10^{12}\text{ FLOPs/second}} \approx 0.061\text{ ms}.$

The roofline estimate uses the larger floor, not their sum: memory and arithmetic can overlap. The ridge is about $989.5/3.35=295.4$ FLOP/byte. At intensity one, peak bandwidth supports roughly 0.34% of dense peak arithmetic throughput. That is a rate comparison, not evidence that the GPU is idle for 99.66% of elapsed time. Thin matrix operations need not attain the advertised Tensor Core peak. The 60 GB payload also leaves only part of the nominal device capacity for caches and working memory; it does not establish model fit.

Verification can amortize weight traffic across several known positions. Let $K$ be draft depth throughout this lesson. A common cached implementation processes one pending committed token plus $K$ proposals, producing $K+1$ next-token distributions. With $m$ input positions, a matrix-vector operation becomes a matrix-matrix operation ($Y=WX$, with $X\in\mathbb{R}^{d\times m}$).

If that pass fetches the weights once and reuses them across all $m$ positions, its weight-only intensity becomes:

$I = \frac{2Pm\text{ FLOPs}}{2P\text{ bytes}} = m\text{ FLOP/byte}.$

For $K=5$ and $m=6$, the compute floor rises to about 0.364 ms while the assumed weight-transfer floor stays 17.91 ms. This explains an opportunity, not a measured verification time. Cache reads/writes, activations, logits, masks, kernel shapes, and communication add costs; tiling need not fetch every weight exactly once. The original papers analyze conditions where short-block scoring stays close to one-step latency.^{[2]Reference 2Accelerating Large Language Model Decoding with Speculative Sampling.https://arxiv.org/abs/2302.01318}

The draft-verify protocol and cache state

The classic chain method alternates sequential drafting with parallel target verification.

Let $M_q$ propose and $M_p$ verify. A hypothetical draft pass fetching 2 GB over the same peak-rate bus has a weight-transfer floor of about 0.6 ms. Five sequential passes have about a 3 ms floor, excluding all other work. Actual drafting includes cache synchronization, kernels, sampling, and any catch-up tokens, so parameter ratio alone cannot predict its cost.

Managing key-value (KV) cache state requires strict bookkeeping across these phases:

1. Distinguish output from computed state. A token is sampled from logits produced before that token is fed through the model. Emitting a token does not compute its KV entries. In ordinary cached drafting, the last sampled proposal may still be pending in the draft cache.
2. Align verification rows. With cached prefix C and pending committed token u, inputs [u, x1, ..., xK] produce distributions [p1, p2, ..., p(K+1)]. Row p1 scores x1; the final row supplies the bonus distribution. Prefix C is not recomputed, though attention can still read its cache.
3. Stop at the first rejection. If x1 through xk survive but x(k+1) fails, discard that token and all its descendants. Sample the correction from that position's residual distribution.
4. Retain valid states only. The target can keep computed KV through xk; rejected-path states must not be reused as the corrected prefix. Each model's valid computed length can differ. The correction must be processed before its KV exists, often as the pending input to the next pass. A token ID is not a cache entry.
5. Handle full acceptance and termination. Emit all $K$ proposals and a target bonus. The bonus is also pending computation. Draft catch-up and cache compaction remain implementation work. Apply EOS, stop sequences, and output limits in emitted order; stop at the first terminating event rather than publishing the rest of a speculative block.

Here is the dependency in miniature. The decimal accumulator is a stand-in for prefix-dependent state, not transformer KV or a serving-cache implementation:

Suppose our committed prefix is The model and the draft proposes serves with cache. Let $q$ represent the draft probability and $p$ represent the target probability at each prefix position.

The first two proposals are accepted unconditionally. The third proposal survives with probability $2/3$, even though the target model assigns it lower probability than the draft did. In this run, a uniform random draw of 0.90 exceeds $2/3$, triggering a rejection for cache.

Exact correction sampling and distribution proof

The following proof applies to the exact classic sampler. It preserves target probabilities, including the target's mistakes; verification is not factual checking. Approximate acceptance heuristics and a changed target checkpoint need separate claims about their output behavior.^{[1]Reference 1Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192[2]Reference 2Accelerating Large Language Model Decoding with Speculative Sampling.https://arxiv.org/abs/2302.01318}

Let $V$ be the vocabulary. For a candidate token $x$ proposed from distribution $q$, the acceptance probability is:

$\alpha(x) = \min\left(1, \frac{p(x)}{q(x)}\right).$

Because $x$ was sampled from $q$, we know $q(x) > 0$, making this ratio well-defined. If $q(x) \le p(x)$, the target desires at least as much probability mass on $x$ as the draft supplied, so the token survives unconditionally ($\alpha = 1$). If $q(x) > p(x)$, the draft overproduced $x$, so we thin its frequency by accepting it with probability $p(x)/q(x)$.

When a token fails the acceptance test, we discard it along with every subsequent proposal in the draft chain. We immediately sample a replacement token from the normalized residual distribution:

$r(x) = \frac{[p(x) - q(x)]_+}{Z}, \quad \text{where } [z]_+ = \max(0, z), \quad Z = \sum_{y \in V} [p(y) - q(y)]_+.$

The term $[p(x) - q(x)]_+$ extracts only the positive probability deficit, identifying tokens where the target demands more probability mass than the draft provided.

Account for both branches

To prove that the emitted token $X$ follows the exact target distribution $p(x)$, trace the two mutually exclusive paths that can generate an output: acceptance or rejection.

Branch 1: Proposal accepted. A token $x$ is proposed with probability $q(x)$ and accepted with probability $\alpha(x)$. Multiplying these gives the probability of proposing and accepting $x$:

$\Pr(\text{proposed } x \text{ and accepted}) = q(x) \cdot \min\left(1, \frac{p(x)}{q(x)}\right) = \min(q(x), p(x)).$

For $q(x)=0$, the proposal-and-accept contribution is zero without evaluating a $0/0$ ratio.

Summing across all vocabulary tokens yields the total probability of accepting any proposal:

$\Pr(\text{accepted}) = \sum_{y \in V} \min(q(y), p(y)).$

Branch 2: Proposal rejected, correction sampled. The probability of rejecting the proposal is the complement of total acceptance:

$\Pr(\text{rejected}) = 1 - \sum_{y \in V} \min(q(y), p(y)) = \sum_{y \in V} p(y) - \sum_{y \in V} \min(q(y), p(y)) = \sum_{y \in V} (p(y) - \min(q(y), p(y))).$

For any real numbers $a$ and $b$, the identity $a - \min(a, b) = \max(0, a - b) = [a - b]_+$ holds. Substituting $p(y)$ and $q(y)$:

$\Pr(\text{rejected}) = \sum_{y \in V} [p(y) - q(y)]_+ = Z.$

The normalization constant $Z$ of the residual distribution is the exact probability of entering the rejection branch.

When rejection happens, we draw token $x$ from the normalized residual $r(x)$. The probability of taking this branch and emitting $x$ is:

$\Pr(\text{rejected and emit } x) = \Pr(\text{rejected}) \cdot r(x) = Z \cdot \frac{[p(x) - q(x)]_+}{Z} = [p(x) - q(x)]_+.$

Combining both branches. By the law of total probability, the marginal probability of emitting token $x$ is the sum of both contributions:

$\Pr(X = x) = \min(q(x), p(x)) + [p(x) - q(x)]_+.$

Applying the algebraic identity $\min(a, b) + \max(0, a - b) = a$ directly confirms:

$\Pr(X = x) = p(x) \quad \forall x \in V.$

The sampling scheme recovers the target distribution exactly. If $p = q$, then $Z = 0$, rejection becomes impossible, and proposals survive with probability 1. Tokens that the draft model never proposes ($q(x) = 0 < p(x)$) remain reachable through the residual branch.

Verify this arithmetic with exact fractions. Notice how resampling directly from raw target $p$ after a rejection distorts the output, over-allocating mass to already-accepted tokens.

Serving transformations and tokenizer alignment

The mathematical proof assumes $p$ and $q$ represent the true probabilities governing the generation event.

Serving-time transformations must be incorporated into $p$ before computing acceptance ratios. If your production serving engine applies temperature scaling, top-$p$ nucleus filtering, top-$k$ truncation, min-$p$ cutoffs, or repetition penalties, $p$ must be the post-transformation, normalized categorical distribution. Comparing pre-transformation logits invalidates the proof.

The two models don't require identical sampling settings. A deterministic proposer, such as prompt lookup decoding, assigns probability $q(x) = 1$ to its proposed candidate and 0 elsewhere. Its acceptance probability simplifies to:

$\alpha(x) = \min\left(1, \frac{p(x)}{1}\right) = p(x).$

If the proposed token is rejected, the residual distribution samples from the remaining target mass with $x$ zeroed out:

$r(y) = \frac{p(y)}{1 - p(x)} \quad \forall y \ne x.$

A stochastic target can safely verify candidates proposed by a deterministic heuristic.^{[1]Reference 1Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

Here's an implementation trap. Suppose the target uses top-2 truncation while the proposer samples without truncation. Only the target distribution receives truncation:

The proof also requires a common token event space. Blindly comparing ID 4012 in one vocabulary with ID 4012 in another is invalid if they mean different tokens. Identical tokenizers are a convenient direct-implementation requirement, not a mathematical necessity: neither every tokenizer uses byte-pair merges nor every cross-tokenizer scheme requires identical segmentation.

Checked September 22, 2026: vLLM's Token-Level Intersection option maps normalized token strings, restricts drafts to shared tokens, and translates their IDs to the target vocabulary. Its documented heterogeneous-vocabulary path supports greedy draft sampling only. A remapped proposer must record probabilities for the distribution it actually samples; renaming IDs alone does not solve general split/merge alignment.^{[4]Reference 4Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}

Implement a complete probability-level round

When every proposal in a depth-$K$ round survives, sample a bonus token directly from the target distribution at position $K+1$. If rejection happens at position $i$, compute the correction using row $p_i$ and row $q_i$ at that exact position.

This CPU implementation constructs a probability-level round over three tokens (cache, latency, batch). The target table changes with preceding context, exposing row-index mistakes. It includes an optional EOS token and output cap, but no transformer, KV cache, GPU parallelism, multi-token stop matcher, or performance benchmark:

Test sequences across multiple positions

A correct distribution on the first token doesn't guarantee that prefix updates are functioning properly. This empirical test runs 60,000 generations and compares the observed frequencies of all nine two-token sequences against the exact joint target probabilities:

This fixed-seed run has maximum absolute frequency error about 0.0037. It is a reproducible smoke check, not a statistical proof or evidence about a GPU sampler. The sequence-distribution proof follows by applying the conditional identity at every emitted prefix, then applying the same stopping rules.

Speedup dynamics and serving load

How many tokens does a speculative round generate on average?

Assume every reached position has the same conditional acceptance probability $\alpha$. An overall average acceptance rate does not establish this assumption. Before termination/output caps, a round emits one correction or bonus plus its accepted prefix. Reaching $i$ accepted proposals then has probability $\alpha^i$.

Summing across all positions gives the expected token yield $\mathbb{E}[N]$ per round:^{[1]Reference 1Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

$\mathbb{E}[N] = 1 + \alpha + \alpha^2 + \dots + \alpha^K = \sum_{i=0}^K \alpha^i = \frac{1 - \alpha^{K+1}}{1 - \alpha}.$

The quotient applies for $\alpha\ne1$. At $\alpha=1$, all proposals survive and $\mathbb{E}[N]=K+1$; the finite sum handles both endpoints.

If $\alpha = 0.8$ and $K = 5$, the system yields:

$\mathbb{E}[N] = \frac{1 - 0.8^6}{1 - 0.8} = \frac{1 - 0.2621}{0.20} \approx 3.69\text{ tokens/round}.$

Measure time relative to one ordinary target decode step. Let $c=T_{\text{draft}}/T_{\text{target}}$ and let $v(K)=T_{\text{verify}}(K)/T_{\text{target}}$ be normalized verification cost. For the simple serial draft/verify model, include sampler, catch-up, transfer, and scheduling time in these costs or add a separate normalized overhead term. Ignoring that extra term, the modeled long-generation speedup is:

$\text{Speedup} \approx \frac{\mathbb{E}[N]}{v(K) + K \cdot c} = \frac{\frac{1 - \alpha^{K+1}}{1 - \alpha}}{v(K) + K \cdot c}.$

If drafting cost is $c = 0.1$ and verification cost remains flat ($v(K) = 1.0$), depth $K = 5$ achieves a speedup of $3.69 / (1.0 + 5 \times 0.1) = 3.69 / 1.50 \approx 2.46\times$. If verification duration grows slightly ($v(K) = 1.0 + 0.05K = 1.25$), speedup declines to $3.69 / 1.75 \approx 2.11\times$.

For $0\le\alpha<1$, expected yield approaches $1/(1-\alpha)$, while positive per-step draft cost keeps growing. At $\alpha=0.8$, the limit is five emitted tokens, not five accepted proposals. A useful depth depends on the measured cost curve, acceptance by position, termination, and load. There is no universal three-to-six-token optimum; parallel draft methods do not even incur the same $Kc$ cost pattern.

Why larger batches can change the answer

Concurrency changes the resources available for verification. It can reduce gains or produce a slowdown, but a batch-size cutoff cannot be inferred from this worksheet.

If weights are reused across $B$ streams, the same approximation gives:

$I_{\text{batch}} = \frac{2P \cdot B}{2P} = B\text{ FLOP/byte}.$

At $B=64$, this is only 64 FLOP/byte, still below the worksheet's dense H100 ridge of about 295.4. It cannot prove compute saturation. Attention/cache traffic and actual GEMM efficiency also matter; request concurrency is not necessarily the scheduler's current decode batch.

For a deliberately restrictive model, assume ordinary decoding and verification are fully compute-limited, each position costs the same, and verification processes $K+1$ positions per stream. Then $v(K)=K+1$. Since $\mathbb{E}[N]\le K+1$, positive sequential draft cost makes the predicted gain below one for $K>0$:

$\text{Speedup}_{\text{equal-cost positions}} \approx \frac{\mathbb{E}[N]}{K+1+Kc}<1\quad(K>0,\ c>0).$

That result follows from the stated assumptions, not from $B\ge32$. Draft weights, states, scratch space, and graph reservations may compete with serving capacity. Shared-feature or parallel proposers can have different costs; adaptive depth may make different decisions at different loads.

There is a concrete counterexample to a universal batching collapse. EAGLE-3's paper reports 1.38x throughput at batch size 64 for LLaMA-Instruct 3.1 8B on one H100 in SGLang v0.4.4, using MT-Bench and a length-three chain without tree verification. The original EAGLE gave 0.99x in that test. These are historical reported results, not measurements of current engines or every workload.^{[5]Reference 5EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testhttps://arxiv.org/abs/2503.01840}

Compare useful delivered throughput, latency, and capacity under the actual offered load. Speculation and continuous batching can be combined; neither wins automatically.

Verification work isn't committed output

Scheduling budgets count work submitted to verification, not just output later accepted. Engines can reserve a fixed window or adapt its size; inspect the installed scheduler's contract.

If each verified stream passes 1 pending committed token plus $K$ proposals, the target forward pass consumes $K+1$ sequence positions per request. A server handling 16 concurrent streams with $K=4$ must evaluate $16 \times (4 + 1) = 80$ positions simultaneously. If the engine's current step budget is capped at 64 positions, that batch can't be scheduled in one pass, even if acceptance rate is near 100%:

Speculative tree architectures and heuristic drafting

An early rejection makes later chain proposals unusable. Under constant conditional acceptance, full-chain survival is $\alpha^K$: at $\alpha=0.70$ and $K=5$, about 0.168. Actual acceptance depends on position and context.

A tree spends more verification work on alternatives instead of only one path. It can improve useful yield, but doesn't guarantee a better cost per emitted token.

Tree attention masks for parallel verification

How does a target model verify multiple competing tree branches in a single parallel forward pass without cross-branch contamination?

For an explicit tree mask, flatten $M$ candidate nodes and construct $M_{\text{tree}}\in\{0,1\}^{M\times M}$. Within candidate columns, node $j$ can attend to an ancestor or itself:

$M_{\text{tree}}[j, i] = \begin{cases} 1 & \text{if candidate } i \text{ is an ancestor of candidate } j \text{ or } i = j, \\ 0 & \text{otherwise.} \end{cases}$

Sibling branches have disallowed attention scores masked to $-\infty$ before softmax, giving zero attention probability. For a model using RoPE, candidate positions use the committed-prefix offset plus tree depth, rather than the flattened index. Other positional schemes and hybrid state require their own compatible handling.

All candidate nodes also attend to the appropriate committed prefix. The mask describes scoring, not an exact stochastic sampler. Greedy decoding can follow the longest path matching target choices; stochastic tree methods need their own proposal ordering and correction algorithm. Arbitrarily picking the longest high-probability path does not preserve $p$.^{[6]Reference 6Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.https://arxiv.org/abs/2401.10774[7]Reference 7EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treeshttps://arxiv.org/abs/2406.16858}

Try a four-node forest attached to the same committed prefix. Nodes 0 and 1 are alternatives; node 2 extends 0 and node 3 extends 1. Self-attention is permitted, but sibling branches stay separate:

This is a visibility worksheet, not a fused attention kernel or a proof of tree sampling. Prefix columns and their position offset are omitted. Candidate input 2's logits predict a continuation after node 2, not the probability of node 2 itself.

Original EAGLE predicts features immediately before the target LM head, using target features and a token sequence shifted one step ahead. The target head turns predicted features into draft token probabilities; EAGLE still proposes tokens. Its paper calls these second-to-top-layer features, rather than a separate earlier transformer block.^{[8]Reference 8EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.https://arxiv.org/abs/2401.15077} EAGLE-2 makes the draft tree context-dependent using draft confidence, with expansion and reranking.^{[7]Reference 7EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treeshttps://arxiv.org/abs/2406.16858}

EAGLE-3 removes the feature-prediction constraint and directly trains token prediction. It fuses low-, mid-, and high-layer target features and simulates multi-step drafting during training. Describing the entire family as feature prediction misses this important change.^{[5]Reference 5EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testhttps://arxiv.org/abs/2503.01840}

Medusa also needs a distinction: Medusa-1 freezes the backbone, while Medusa-2 trains it jointly with the heads. Joint training can change the target itself. The paper's typical acceptance heuristic explicitly relaxes distribution matching for stochastic generation; the exact residual proof above does not establish its losslessness.^{[6]Reference 6Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.https://arxiv.org/abs/2401.10774}

Prompt lookup requires no extra model training. A common scheme matches a trailing token n-gram against prompt text and proposes the following span. Some engines also search generated-text caches. It can be useful when outputs repeat available text, but searching, maintaining indexes, and verifying candidates still cost memory and time. A copied span remains a proposal, not a trusted legal or factual answer.^{[9]Reference 9N-Gram Speculationhttps://docs.vllm.ai/en/latest/features/speculative_decoding/n_gram/[10]Reference 10Speculative Decodinghttps://docs.sglang.io/docs/advanced_features/speculative_decoding}

Implementation snapshot, September 22, 2026: vLLM documents MTP and parallel PARD drafting in addition to EAGLE and n-grams; these are distinct trained/model-family paths, not switches that make any checkpoint compatible.^{[11]Reference 11MTP (Multi-Token Prediction)https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/[12]Reference 12Parallel Draft Modelshttps://docs.vllm.ai/en/latest/features/speculative_decoding/parallel_draft_model/} SGLang documents EAGLE-3, standalone drafting, MTP, and DFlash block verification, among other methods.^{[10]Reference 10Speculative Decodinghttps://docs.sglang.io/docs/advanced_features/speculative_decoding} This is a method map, not a speed ranking.

Record engine version, target/draft revisions, quantization, attention backend, parallel topology, and sampler settings. Check returned-logprob, grammar, penalties, multimodal, graph, and scheduler compatibility for that exact method. vLLM documents finite-precision/logprob variability; neither seed equality nor greedy equality proves arbitrary stochastic correctness.^{[4]Reference 4Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}

Measure serving speedup with paired benchmarks

Speculative decoding performance depends heavily on your specific serving environment. Validate improvements using a strictly controlled paired experiment before rolling out to production.

Keep the target model checkpoint, quantization format, prompt distribution, sampling parameters, and server hardware topology identical across both arms. Record non-speculative baseline latency, then run the speculative configuration under matching offered load.

Track the following production metrics:

• Client TTFT and completion latency: Include queueing, target prefill, draft initialization, and delivery. Speculation accelerates generation work; it does not remove prompt processing. Measure any first-token regression against the product requirement.
• Delivered token gaps: Record client timestamps and tail pauses. Tokens from one round may arrive in a burst followed by a long gap; average server time per token does not describe that experience. Separate reasoning tokens from visible answer tokens when relevant.
• Accepted proposals and emitted yield: Log both distributions. Yield includes a correction or bonus and can be reduced by termination. Three accepted proposals is not a universal health threshold.
• Acceptance by reached position: Count accepted/reached at each depth and segment by task, context, and sampling mode. A first-position rate alone cannot decide whether drafting is profitable.
• Useful capacity under load: Sweep representative offered request rates and scheduler batches, beyond 64 if relevant. Track delivered throughput, latency SLO attainment, memory peaks, failures, and queue growth. A sampled crossover has uncertainty and can change with workload or adaptive depth.

Use repeated paired prompt sets, report uncertainty, and account for stochastic output lengths. For a fixed-token timing experiment, state the token cap and EOS treatment; for completed user tasks, report the actual completion-length distribution. Track extra verification positions and draft work separately from useful output. A faster isolated stream is not proof of lower cost per completed task or greater SLO-qualified serving capacity.

Follow-up questions

How do position-dependent acceptance rates affect expected length?

Let $a_i$ denote the conditional probability of accepting the $i$-th candidate given that candidates 1 through $i-1$ were accepted. The expected emitted tokens per round is:

$\mathbb{E}[N] = 1 + a_1 + a_1 a_2 + a_1 a_2 a_3 + \dots + \prod_{i=1}^K a_i.$

Measure acceptance rates strictly from reached positions, not all drafted positions. If position 1 acceptance drops on challenging reasoning prompts, expected length collapses rapidly across all downstream positions.

Even the arithmetic mean of conditional rates loses crucial information. Compare these depth-two fixtures:

How do you isolate a sampler defect from GPU kernel precision shifts?

First, isolate the probability sampler on synthetic fixed probability distributions across CPU and GPU. Test edge cases including complete agreement ($p=q$), zero overlap, and scripted rejections at every position. If fixed tables match, compare target model logits produced during serial decoding against logits produced during batched verification.

Model-only discrepancies narrow the investigation to masks, position/logit alignment, cache state, preprocessing, and numerical behavior; they do not identify one cause. Quantized kernels, deterministic reduction-order differences, and batch-dependent rounding can matter without random kernel execution. Check serial/verification logits under matched inputs and tolerances, then test the complete sampler, stopping, and delivery paths. A finite frequency test cannot prove equality across every prompt or tiny-probability event.

What strong answers show

• State the assumptions behind weight-only intensity and distinguish a roofline floor from measured latency or idle time.
• Derive the residual identity for actual conditional sampling distributions; qualify numerics, termination, tokenizer mapping, and approximate acceptance.
• Align verification logits and retain only valid computed states. Explain why a correction or bonus remains pending until processed.
• Compare yield against measured drafting, verification, and scheduling cost under actual load, including burst delivery and serving capacity.
• Distinguish original EAGLE, EAGLE-3, Medusa variants, MTP, parallel drafting, and lookup methods without turning one benchmark into a universal ranking.

Check your understanding