Skip to content
Skip to lesson
Back to Topics
LearnInference & Production ScaleScaling LLM Inference
🚀HardInference Optimization

Scaling LLM Inference

Size a serving replica from memory bandwidth and KV residency, then pick the batch, precision, and speculation the latency target can afford.

48 min read
Learning path
Step 144 of 199 in the full curriculum

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Continuous batching can refill decode slots as soon as a request finishes. A full queue can still hide a capacity failure: once the scheduler is willing to admit more work, what runs out first on the GPU serving a large language model (LLM)?

A code-assistant pool makes the constraint concrete. Short completions and long patch explanations share one model replica, and each request advances one at a time during ordinary decode.

Each decode step reads model weights from and consults a growing . For small-batch dense-model decode, moving those bytes often costs far more time than the arithmetic. Memory capacity separately limits how many concurrent streams fit in VRAM.

Start with an 8B-class replica on one GPU. As we add requests, its weights stay fixed but its KV occupancy grows. The larger 70B architecture stress-tests the same calculation. The CPU examples below check arithmetic, scheduling, and sampling behavior; they don't measure GPU throughput or run a live serving engine.

Decode sits under a bandwidth roof

You already know the two phases from inference mechanics. reads known prompt tokens in parallel and is compute-heavy.

emits one new token per request step. This sequential dependency is why a request can have prompt processing behind it but still need hundreds of serial iterations to finish.

Users wait for queueing, prompt processing, and delivery before the first token: together these make up . During decode they experience , also called time between tokens (TBT). For a response with at least two tokens, averages the interval from first to last over the remaining tokens. A steady mean can easily mask a terrible ITL.[1]

The root physical constraint is the split in arithmetic intensity. measures FLOPs performed per byte loaded from HBM.

Prefill reuses each weight across many prompt positions in matrix multiplications (GEMMs). One-request decode makes the dense linear operations matrix-vector products (GEMVs); batched decode uses thin GEMMs. Attention over cached history adds separate work.

For a dense model with PP parameters, use roughly 2P2P FLOPs per token for the parameter-dependent linear work. This omits context-dependent attention and counts multiply-add as two FLOPs. If one step streams the weights once across batch size BB and weight reads dominate traffic, the simplified intensity is:

Idecode≈2×P×BP×Pbytes=2BPbytesI_{\text{decode}} \approx \frac{2 \times P \times B}{P \times P_{\text{bytes}}} = \frac{2 B}{P_{\text{bytes}}}

For FP16 precision (Pbytes=2P_{\text{bytes}} = 2), the operational intensity simplifies to Idecode≈BI_{\text{decode}} \approx B FLOPs per byte.

The bounds performance by the smaller of a compatible compute peak and times intensity.[2][3] Below the ridge, the bandwidth roof is lower; above it, the compute roof is lower. Reaching either roof still requires suitable kernels and achieved hardware rates.

For H100 SXM, use peak HBM bandwidth of 3.35 TB/s and about 989.5 TFLOP/s of dense FP16/BF16 Tensor Core compute. NVIDIA's 1,979 figure includes sparsity and must be halved for this dense comparison.[4] The illustrative ridge is:

Iridge=989.5×1012 FLOP/s3.35×1012 bytes/s≈295.4 FLOPs/byteI_{\text{ridge}} = \frac{989.5 \times 10^{12}\text{ FLOP/s}}{3.35 \times 10^{12}\text{ bytes/s}} \approx 295.4\text{ FLOPs/byte}

At B=1B=1, the weight-only approximation gives 1 FLOP/byte and a 3.35 TFLOP/s bandwidth roof, about 0.34% of that compute peak. This compares rates; it does not mean the tensor cores are idle for 99.66% of elapsed time. Device counters and a trace are needed to describe utilization.

Log₂-spaced H100 SXM roofline. Bandwidth is 3.35 TB/s and dense FP16 compute is 989.5 TFLOP/s; their curves meet at 295.4 FLOP/byte. The plotted ceilings are 3.35, 107.2, and 428.8 TFLOP/s at intensity 1, 32, and 128, then 989.5 TFLOP/s at the ridge. The lower of the byte-transfer and arithmetic ceilings limits the rate. I approximately equals batch size only when FP16 weights dominate; KV traffic changes intensity.
HBM bytes set the traffic ceiling, and dense FP16 operations set the compute ceiling. At each intensity, the lower rate bounds performance. These are H100 ceilings, not measured kernel rates. With weight-only FP16 traffic, I ≈ B; KV bytes and attention work change intensity. At I=1, 0.34% compares ceilings, not elapsed idle time.

Llama 3 8B is an 8B-parameter dense model.[5] An copy of those weights occupies about 16 GB (8×109×28\times10^9\times2 bytes).

If one uncached decode step streams that entire footprint for one active token, the weight-read upper bound on H100 SXM is 3,350/16≈2093{,}350/16\approx209 single-token steps per second. This is an idealized bandwidth upper bound; production throughput also depends on KV traffic, attention calculations, and runtime overhead.

Under that full-read assumption, dynamic KV cache traffic, attention calculations, and runtime overhead push the achievable rate lower. Weight quantization, speculative decoding, and batching change the bound. Use it as an immediate diagnosis: if a trace saturates HBM while tensor cores sit idle, buying more FLOPs won't speed up decode.

decode-bandwidth-lower-bound.py
parameters = 8_000_000_000  # Llama 3 8B-class dense count
bytes_per_parameter = 2  # FP16
h100_sxm_hbm_gb_s = 3_350  # decimal GB/s, NVIDIA H100 SXM

weight_gb = parameters * bytes_per_parameter / 1_000_000_000
ideal_steps_per_second = h100_sxm_hbm_gb_s / weight_gb

print(f"FP16 weight footprint: {weight_gb:.1f} GB")
print(f"H100 SXM weight-read upper bound: {ideal_steps_per_second:.1f} single-token steps/s")
print("Under this full-read model, extra traffic and overhead can only lower the bound.")
Output
FP16 weight footprint: 16.0 GB
H100 SXM weight-read upper bound: 209.4 single-token steps/s
Under this full-read model, extra traffic and overhead can only lower the bound.

KV bytes cap the batch you can form

Batching only helps if extra requests fit. The KV cache lesson derived the payload formula: each token stores a Key vector and a Value vector at every layer and KV head.

KV_bytes_per_token=2×Nlayers×Nkv_heads×dhead×Pbytes\text{KV\_bytes\_per\_token} = 2 \times N_{\text{layers}} \times N_{\text{kv\_heads}} \times d_{\text{head}} \times P_{\text{bytes}}

Llama 3 uses with 8 KV heads at 8B, 70B, and 405B.[5][6] That head count is why KV state is much smaller than a full multi-head attention cache, but it still grows linearly with layers, sequence length, and concurrency.

For Llama 3 70B in FP16 (Nlayers=80,Nkv_heads=8,dhead=128,Pbytes=2N_{\text{layers}} = 80, N_{\text{kv\_heads}} = 8, d_{\text{head}} = 128, P_{\text{bytes}} = 2), one single token demands 2×80×8×128×2=327,6802\times80\times8\times128\times2=327{,}680 bytes, or 320 KiB. Multiplying by 8,192 tokens yields 2.5 GiB per request. Sixty-four concurrent 8K streams require 160 GiB of KV cache alone, before loading a single weight byte. An 80 GB H100 can't hold that batch: a plan to keep cranking batch size until the roofline saturates crashes into VRAM residency first.

The smaller 8B model fits on a single accelerator. Llama 3 8B has 32 layers, 8 KV heads, and dhead=128d_{\text{head}} = 128, requiring 128 KiB per token. A 4,096-token request takes exactly 0.5 GiB.[5]

For independent, unshared full-attention histories of the same length LL, a fixed pool gives a payload-only batch ceiling:

Bmax(L)=⌊MKV_poolKV_bytes_per_request(L)⌋=⌊MKV_pool2×L×Nlayers×Nkv_heads×dhead×Pbytes⌋B_{\text{max}}(L) = \left\lfloor \frac{M_{\text{KV\_pool}}}{\text{KV\_bytes\_per\_request}(L)} \right\rfloor = \left\lfloor \frac{M_{\text{KV\_pool}}}{2 \times L \times N_{\text{layers}} \times N_{\text{kv\_heads}} \times d_{\text{head}} \times P_{\text{bytes}}} \right\rfloor

Assume a 51.60 GiB pool, derived from an explicit byte budget below. The 8B dimensions give these capacities. For the longer rows, use a checkpoint such as Llama 3.1 8B that supports 128K.[5] The original April Llama 3 8B has an 8K limit.[7] Matching memory geometry doesn't extend a model's supported context.

Sequence length (LL)KV per request (FP16)Max concurrent streams (BmaxB_{\text{max}})Concurrency drop
4,096 tokens (4K)0.50 GiB103 requestsBaseline
16,384 tokens (16K)2.00 GiB25 requests-76%
32,768 tokens (32K)4.00 GiB12 requests-88%
65,536 tokens (64K)8.00 GiB6 requests-94%
131,072 tokens (128K)16.00 GiB3 requests-97%

Before flooring, payload capacity scales as 1/L1/L. The 128K row shows room for three streams under the stated pool assumption; measure throughput separately. A fourth independent full-payload request can't join that snapshot; it must wait, be rejected, or cause another request to be preempted. A serving policy chooses among those outcomes.

Five equal 51.60 GiB tracks divide into one cell per unshared KV history. The rows show 103 requests of 0.50 GiB at 4K, 25 of 2 GiB at 16K, 12 of 4 GiB at 32K, 6 of 8 GiB at 64K, and 3 of 16 GiB at 128K. Unused tails show free space. A fourth 128K history needs 16 GiB when only 3.60 GiB remains.
At 128K, three 16 GiB histories use 48 GiB, leaving 3.60 GiB; a fourth cannot fit. The fixed 51.60 GiB pool counts equal, unshared payload only, not safe admission or throughput. Growth, page rounding, sharing, runtime memory, and latency targets need margin. Long rows require a checkpoint that supports 128K.
llama70b-kv-residency.py
def kv_cache_gib(
    batch_size: int,
    seq_len: int,
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    dtype_bytes: int = 2,
) -> float:
    dimensions = (batch_size, seq_len, num_layers, num_kv_heads, head_dim, dtype_bytes)
    if any(type(value) is not int or value <= 0 for value in dimensions):
        raise ValueError("all dimensions and dtype_bytes must be positive integers")
    total_bytes = (
        2 * batch_size * seq_len * num_layers * num_kv_heads * head_dim * dtype_bytes
    )
    return total_bytes / (1024 ** 3)

one_8k = kv_cache_gib(1, 8_192, 80, 8, 128)
batch_64 = kv_cache_gib(64, 8_192, 80, 8, 128)

print(f"Llama 3 70B, one 8K request: {one_8k:.1f} GiB")
print(f"64 active 8K requests: {batch_64:.0f} GiB")
print("single-request estimate correct:", one_8k == 2.5)
print("64-request estimate correct:", batch_64 == 160.0)
assert kv_cache_gib(1, 4_096, 32, 8, 128) == 0.5
assert kv_cache_gib(2, 8_192, 80, 8, 128) == 2 * one_8k
try:
    kv_cache_gib(1, -1, 80, 8, 128)
except ValueError:
    pass
else:
    raise AssertionError("negative sequence length accepted")
Output
Llama 3 70B, one 8K request: 2.5 GiB
64 active 8K requests: 160 GiB
single-request estimate correct: True
64-request estimate correct: True

That 160 GiB figure assumes every request occupies a live 8K payload. A max-context slab reservation is far worse: it reserves memory for tokens that never arrive.

PagedAttention allocates fixed-size blocks (e.g. 16 tokens) as needed and maps logical order through a block table.[8] Exclusively owned pages become reusable after their readers finish; shared or cached pages need separate ownership/eviction accounting. Paging avoids maximum-length reservations without shrinking live-token payload. A partial last block wastes at most fifteen slots here, not universally less than 4%: a one-token history wastes 15/1615/16 slots, or 93.75%93.75\%.

Two allocation snapshots hold the same eleven live blocks: A has two, B five, C three, and D one. Four maximum-length slabs reserve 32 blocks, leaving 21 unused. A separate 15-block paged pool holds the same blocks in interleaved physical order, with four available. Logical A0 and A1 map to nonadjacent physical slots.
Both states hold eleven populated blocks. Slabs reserve 21 of 32; the separate paged pool has four free of 15 and maps A's logical order to nonadjacent locations. Paging changes allocation; this fixture makes no throughput claim.[8]

The throughput, latency, cost triangle

Once bandwidth and KV residency are in view, "maximize tokens per second" is not a complete objective. Start with the product promise: interactive users care about TTFT and p95 ITL, while offline batch jobs care about completion throughput and cost per token.

Larger batches amortize weight reads. They also bring more KV history and arithmetic into each iteration. The roughly 4.8 ms calculation is an ideal weight serialization floor under a full-read, peak-bandwidth assumption. Decode duration also depends on per-request KV traffic, arithmetic, cache reuse, launches, and queueing, especially at batch sixteen.

Try an explicit full-read thought experiment: 16 GB of weights plus 0.5 GiB of KV per active 4K history, all read once from HBM per step at peak 3.35 TB/s. Exclude attention FLOPs, writes, launches, cache reuse, and queueing. The assumed byte cost already makes step duration rise with batch size:

Three aligned plots use the same batch positions. With 16 decimal GB of weights and 0.536870912 GB of KV per 4K request, bytes per step rise from 16.54 to 50.36 GB as batch grows from 1 to 64. At assumed 3.35 TB/s those reads give 4.94 to 15.03 ms serialization floors and 202.6 to 4257.4 aggregate tokens per second. Linear intensity is 1.0, 6.3, 15.4, and 20.3 FLOPs per byte. These are assumed floors, not measured latency.
The same byte total sets each serialization floor, aggregate rate, and linear intensity. The fixture excludes attention FLOPs and writes, launches, cache reuse, and queueing; measure the workload before forecasting TTFT or ITL.
kv-traffic-changes-batch-scaling.py
weight_bytes = 16_000_000_000
kv_per_request = 2 * 4096 * 32 * 8 * 128 * 2
peak_bytes_per_second = 3.35e12
linear_flops_per_token = 2 * 8_000_000_000

for batch in (1, 8, 32, 64):
    bytes_read = weight_bytes + batch * kv_per_request
    seconds = bytes_read / peak_bytes_per_second
    intensity = batch * linear_flops_per_token / bytes_read
    print(f"B={batch}: {seconds * 1000:.2f} ms, {batch / seconds:.1f} tokens/s, {intensity:.1f} linear FLOPs/byte")
print("Assumed full weight+KV reads at peak bandwidth; ideal serialization floor.")
Output
B=1: 4.94 ms, 202.6 tokens/s, 1.0 linear FLOPs/byte
B=8: 6.06 ms, 1320.5 tokens/s, 6.3 linear FLOPs/byte
B=32: 9.90 ms, 3230.9 tokens/s, 15.4 linear FLOPs/byte
B=64: 15.03 ms, 4257.4 tokens/s, 20.3 linear FLOPs/byte
Assumed full weight+KV reads at peak bandwidth; ideal serialization floor.

At B=64, the parameter-dependent work has intensity about 20.3 FLOPs/byte in this model, rather than the weight-only estimate of 64. Actual attention adds FLOPs, and actual layouts/cache reuse change traffic. Neither formula is a substitute for profiling. The useful question is which bytes and operations your estimate omitted.

In a live endpoint, inspect three additional pressures:

  • KV reads and arithmetic can increase iteration duration as membership and history grow.
  • Mixed prefills, transfers, and client buffering can change token gaps.
  • Offered arrivals beyond sustainable service capacity build queues unless work is rejected or expired. KV exhaustion may cause waiting or preemption, depending on policy.
ObjectiveTypical leverTrade-off
Throughput (tokens/s across replica)Amortize reads with effective batchesMore resident history/work can increase per-step latency
Lower latency (TTFT, p95 ITL)Bound work, queue wait, and prompt interferenceMay leave capacity unused; measure the arrival mix
Lower cost per useful tokenSustain quality/-qualified output per billed intervalInclude idle time, retries, other workers, and fixed costs

The GPU-only cost component is hourly GPU spend divided by sustained output. It excludes CPU/network/storage and other service costs. Use useful, quality-qualified output if that is the product's objective:

cost per token=GPU $ per hoursustained tokens per second×3600\text{cost per token} = \frac{\text{GPU \$ per hour}}{\text{sustained tokens per second} \times 3600}

Treat $3.00/hour as an arithmetic , not a cloud quote. Suppose a replica sustains 2,500 output tokens/s across active requests over the billed interval, including prompt processing and idle periods:

  • Tokens per hour = 2,500×3,600=9,000,0002{,}500\times3{,}600=9{,}000{,}000
  • Cost per million tokens = 3.009,000,000×1,000,000≈0.33\frac{3.00}{9{,}000{,}000}\times1{,}000{,}000\approx0.33 dollars ($0.33)

At 250 output tokens/s, it rises to $3.33. Latency constraints can justify that gap, but throughput alone doesn't identify the cause: low demand or an inefficient runtime can also produce it.

cost-per-million-tokens.py
import math

def cost_per_million(hourly_cost: float, sustained_tps: int) -> float:
    if (not math.isfinite(hourly_cost) or hourly_cost < 0
            or type(sustained_tps) is not int or sustained_tps <= 0):
        raise ValueError("cost must be finite/nonnegative; throughput a positive integer")
    cost = hourly_cost / (sustained_tps * 3600) * 1_000_000
    if not math.isfinite(cost):
        raise ValueError("reported cost overflowed")
    return cost

well_batched = cost_per_million(3.00, 2_500)
starved = cost_per_million(3.00, 250)

print(f"2,500 tokens/s: ${well_batched:.2f} per million tokens")
print(f"250 tokens/s: ${starved:.2f} per million tokens")
print(f"cost multiplier: {starved / well_batched:.0f}x")
Output
2,500 tokens/s: $0.33 per million tokens
250 tokens/s: $3.33 per million tokens
cost multiplier: 10x

A missing price encoded as NaN must not become a reported cost. The helper rejects nonfinite financial inputs instead of passing them into a dashboard:

missing-price-is-not-a-cost.py
for malformed_price in (float("nan"), float("inf")):
    try:
        cost_per_million(malformed_price, 250)
    except ValueError:
        pass
    else:
        raise AssertionError("nonfinite price became a reported cost")
print("missing/infinite prices rejected")
Output
missing/infinite prices rejected

Optimized kernels or schedulers can improve throughput and latency together. A Pareto frontier contains measured configurations where improving one objective requires worsening another, under the same workload and comparison contract. There is no universal B=32–64 sweet spot. Find feasible configurations under your latency and quality targets, then compare their cost.

Operating pointBatch pressureCost per tokenLatency profileWorkload fit
Latency-firstLimit mixed-step work and waitingCan be higher if fewer tokens amortize spendMust meet measured TTFT/ITL targetsChat, autocomplete
BalancedFit admission to the measured workloadCompare feasible cost pointsCheck tails by request classAssistant endpoints
Throughput-firstKeep sustained useful work availableCan be lower with good amortizationAccept the workload's completion targetsOffline batch, evaluations

Production tip: Watch KV occupancy, prefill backlog, queue age, and HBM bandwidth together. High KV occupancy plus preemptions suggests a residency limit. Low KV occupancy and idle compute may mean low demand, a CPU bottleneck, or bandwidth-bound decode; they don't by themselves prove that a larger batch is available.

Batching raises arithmetic intensity

The previous chapter treated as slot reuse: when request A finishes, D can enter at the next decode iteration instead of waiting for C's long answer.[9] Slot reuse is only half the story. One decode step still streams the weight tensor roughly once for the whole batch. More active tokens on that step means more useful FLOPs per byte, which is how serving climbs the roofline without changing the model.

Static batching fights that climb. If three assistant requests need 2, 2, and 5 remaining tokens, a request-level batch of size 3 spends later steps on padding or idle slots. Iteration-level scheduling keeps the weight read attached to live work.

Predict the comparison before running it: finished slots should admit new work sooner. Both policies receive the same five requests with remaining lengths [2, 2, 5, 3, 1] and three slots. This sketch counts decode iterations as idealized full-weight reads, not milliseconds. Prompts are assumed already processed; there are no prefill costs, KV limits, arrivals, or kernel timing.

batch-amortizes-weight-reads.py
from collections import deque

lengths = [2, 2, 5, 3, 1]
static_steps = max(lengths[:3]) + max(lengths[3:])
static_tokens = sum(lengths)

queue = deque([2, 2, 5, 3, 1])
active: list[int] = []
max_batch = 3
weight_reads = 0
emitted = 0

while queue or active:
    while queue and len(active) < max_batch:
        active.append(queue.popleft())
    if not active:
        break
    weight_reads += 1
    nxt: list[int] = []
    for remaining in active:
        remaining -= 1
        emitted += 1
        if remaining > 0:
            nxt.append(remaining)
    active = nxt

print(f"static batch: {static_tokens} tokens in {static_steps} weight reads")
print(f"continuous batch: {emitted} tokens in {weight_reads} weight reads")
print(f"tokens per weight read (static): {static_tokens / static_steps:.2f}")
print(f"tokens per weight read (continuous): {emitted / weight_reads:.2f}")
assert emitted == static_tokens == 13
assert (static_steps, weight_reads) == (8, 5)
Output
static batch: 13 tokens in 8 weight reads
continuous batch: 13 tokens in 5 weight reads
tokens per weight read (static): 1.62
tokens per weight read (continuous): 2.60

Both policies emit 13 tokens. takes five iterations for the first group, then three for the second. Continuous batching starts the last two requests after iteration two and finishes everything in five iterations. The schedule reduces idealized weight reads from eight to five. Measure step duration across changing batch membership and context length before estimating throughput. Admitting past the cache budget can trigger eviction or preemption.

Sharing, chunking, and phase isolation

Three more levers change the same budget without training a new model. Each targets a different source of pressure: repeated prompt work, prompt interference, or movement between phase-specific workers.

Reuse identical prefixes

Many assistant requests start from the same system prompt. Prefix caching is the lookup that finds reuse across independent requests.

Copy-on-write protects shared state: if two continuations share a partially filled block, a writer first clones that block before appending to it.[8] Existing prefix tokens aren't edited. A full shared block stays immutable, and new tokens go into private tail blocks. Independent prefix caches often reuse only full blocks, so they needn't clone a partial block on every hit.

For Llama 3 8B, an identical 4K prefix has a 0.5 GiB KV payload stored once if sharing is supported. Each later session still needs metadata, private continuation KV, and decode attention over that prefix. Reuse requires identical effective model state, token IDs and positions, plus an appropriate tenant-isolation policy. It saves prefix computation and duplicate storage, not all subsequent attention work.

Three states show copy-on-write for A and B. Both first map to partial block 7 containing p, q, r and an empty slot, with reference count two. B clones block 7 into block 12; each private block then has one reference. A appends a to block 7 and B appends b to block 12, preserving the shared prefix values.
Clone the partially filled physical page before either private write. A shared partial block still needs isolation when both next tokens match; full shared prefix blocks remain immutable, and prefix lookup is a separate operation.

Bound prompt interference

A 10K RAG prompt on the same worker as live streams can increase token gaps. SARATHI studies combining chunked prefills with decode work.[10] Checked October 10, 2026: V1 enables chunked prefill when possible, prioritizes pending decode work, then spends remaining max_num_batched_tokens budget on prompt chunks.[11]

Chunking doesn't add HBM or guarantee smoother delivery. Smaller mixed iterations can reduce interference, but extra boundaries/context work can increase prefill cost. Remeasure TTFT and ITL together; details live in the scheduling chapter.

Separate phases when transfer is worth paying

Splitwise and DistServe move prefill and decode onto separate pools when avoided interference or better resource allocation pays for KV transfer.[12][1] This changes worker placement, not payload size.

For a Llama 3 70B 8K request, payload is 2.5 GiB. At an assumed effective one-way rate of 200 GiB/s, serializing the whole payload takes at least 12.5 ms before protocol and scheduling. This isn't a named interconnect specification. Overlap, sharding, contention, and already-cached blocks change the transfer's critical-path cost. In a simple non-overlapped comparison, the split must save more time than transfer plus coordination adds.

kv-handoff-lower-bound.py
kv_cache_gib = 2.5  # Llama 3 70B, one 8K request
interconnect_gib_s = 200  # binary GiB/s

ideal_transfer_ms = kv_cache_gib / interconnect_gib_s * 1000
mixed_ms = (kv_cache_gib * 1024**3) / (200 * 1_000_000_000) * 1000

print(f"KV state to transfer: {kv_cache_gib:.1f} GiB")
print(f"ideal one-way floor at {interconnect_gib_s} GiB/s: {ideal_transfer_ms:.1f} ms")
print(f"mixed GiB / decimal-GB/s floor (for comparison): {mixed_ms:.1f} ms")
print("Queueing saved must exceed transfer plus protocol and scheduling overhead.")
Output
KV state to transfer: 2.5 GiB
ideal one-way floor at 200 GiB/s: 12.5 ms
mixed GiB / decimal-GB/s floor (for comparison): 13.4 ms
Queueing saved must exceed transfer plus protocol and scheduling overhead.

Bandwidth sets how hard you can push the batch. KV residency sets how many requests that batch can contain. The operating point is where those two ceilings meet the product's latency SLO.

Speculative decoding buys tokens per weight read

Batching amortizes one target-model weight read across concurrent requests. applies the same insight within a single stream: one target weight read can verify several sequential future tokens.[13]

In the classic algorithm, a draft model proposes γ\gamma candidate tokens. Target verification supplies γ+1\gamma+1 conditional distributions: one for each proposal and one for a possible concluding token.[13] A causal forward pass can evaluate the drafted prefixes together; cached logits and runtime layout determine how many new input positions it schedules. Verification cost depends on attention/cache work and backend efficiency, so measure it against ordinary one-token decode.

Let c=Tdraft/Ttargetc=T_{\text{draft}}/T_{\text{target}} be draft-step cost relative to ordinary target decode. Measure it; parameter-count ratios don't determine it. Let vv be verification plus remaining cycle overhead in the same units. For serial draft generation followed by verification:

Tcycle=(γc+v)TtargetT_{\text{cycle}} = (\gamma c + v) T_{\text{target}}

Acceptance decisions follow proposal order, using scores already computed by verification. For a proposal actually sampled from qq, use target distribution pp conditioned on the same prefix. Both distributions include the intended sampling transformations, such as temperature or top-p.[13] Accept with probability:

αi=min⁡(1,p(xi)q(xi))\alpha_i = \min\left(1, \frac{p(x_i)}{q(x_i)}\right)

On the first rejection at position ii, the remaining draft tokens are discarded, and the engine immediately samples a replacement token from the normalized positive residual distribution:

presidual(x)=max⁡(0,p(x)−q(x))∑x′max⁡(0,p(x′)−q(x′))p_{\text{residual}}(x) = \frac{\max(0, p(x) - q(x))}{\sum_{x'} \max(0, p(x') - q(x'))}

If all proposals survive, sample an extra token from the target distribution conditioned on the accepted prefix. The ideal algorithm emits 1 through γ+1\gamma+1 tokens before stopping/truncation and preserves the intended target distribution, not an identical seeded sequence. Real finite-precision implementations require their own numerical and sampling checks.

The paper's geometric expectation assumes acceptance rates are independent and identically distributed, with mean α\alpha.[13] Under that assumption, the probability that the first ii proposals all survive is αi\alpha^i. A fleet-wide mean alone doesn't establish those survival probabilities. Add the concluding correction/bonus sample:

E[N]=1+∑i=1γαi=∑i=0γαi=1−αγ+11−α\mathbb{E}[N] = 1 + \sum_{i=1}^\gamma \alpha^i = \sum_{i=0}^\gamma \alpha^i = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}

For α=1\alpha=1, the limit is γ+1\gamma+1. Compare the resulting expected rate with ordinary target decode, assuming long enough generations and the costs above:

Speedup=E[N]γc+v\text{Speedup} = \frac{\mathbb{E}[N]}{\gamma c + v}

For an assumed v=1v=1, γ=4\gamma=4, c=0.10c=0.10, and iid acceptance mean α=0.80\alpha=0.80:

E[N]=1−0.851−0.8=0.672320.2=3.36 tokens\mathbb{E}[N] = \frac{1 - 0.8^5}{1 - 0.8} = \frac{0.67232}{0.2} = 3.36\text{ tokens} Speedup=3.36164×0.10+1=3.36161.40≈2.40×\text{Speedup} = \frac{3.3616}{4 \times 0.10 + 1} = \frac{3.3616}{1.40} \approx 2.40\times

The expectation predicts 3.36 tokens per cycle costing 1.40 baseline-step units, or a 2.40×2.40\times rate ratio. Measure cycles on the target engine to estimate the observed speedup.

Now consider the failure mode: if the draft model struggles on code or dense reasoning, and acceptance drops to α=0.30\alpha = 0.30 while draft cost is c=0.15c = 0.15:

E[N]=1−0.351−0.3=0.997570.7≈1.425 tokens\mathbb{E}[N] = \frac{1 - 0.3^5}{1 - 0.3} = \frac{0.99757}{0.7} \approx 1.425\text{ tokens} Speedup=1.4254×0.15+1=1.4251.60≈0.89×\text{Speedup} = \frac{1.425}{4 \times 0.15 + 1} = \frac{1.425}{1.60} \approx 0.89\times

The modeled token rate is about 11% lower; time per token is about 12% higher, since it scales as the reciprocal. The break-even condition for this serial cost model is E[N]>γc+v\mathbb{E}[N]>\gamma c+v. Acceptance, span length, and verification overhead must be measured together.

One illustrative gamma-four cycle drafts x1 through x4. A target pass scores p1 through p5 together, while acceptance decisions inspect x1, x2, x3, then reject x4. The residual sample x4-prime replaces the rejection, so this cycle emits four tokens. Separately, iid alpha 0.8 with draft cost 0.1 and verify cost 1T gives expected 3.3616 tokens per 1.4T cycle and a 2.40x rate ratio; other assumed costs give 1.40x or 0.89x.
The upper trace is one illustrative gamma-four outcome. The lower rows model an iid acceptance assumption and separate draft/verify costs; the article's seeded gamma-two CPU fixture remains a different example. Measure the target engine.
speculative-accept-reject.py
import random
import math

def validate_dist(dist: list[float], vocab: int) -> None:
    if len(dist) != vocab or any(not math.isfinite(x) or x < 0 for x in dist):
        raise ValueError("invalid distribution entries")
    if not math.isclose(sum(dist), 1.0, abs_tol=1e-9):
        raise ValueError("distribution must sum to one")

def sample(dist: list[float], rng: random.Random) -> int:
    cutoff = rng.random() * sum(dist)
    running = 0.0
    for i, mass in enumerate(dist):
        running += mass
        if cutoff < running:
            return i
    return max(i for i, mass in enumerate(dist) if mass > 0)

def residual_sample(p: list[float], q: list[float], rng: random.Random) -> int:
    leftover = [max(pt - qt, 0.0) for pt, qt in zip(p, q)]
    total = sum(leftover)
    if total <= 0:
        raise ValueError("rejection requires positive residual mass")
    return sample(leftover, rng)

def speculative_step(
    draft_tokens: list[int],
    draft_dists: list[list[float]],
    target_dists: list[list[float]],
    rng: random.Random,
) -> list[int]:
    k = len(draft_tokens)
    if len(draft_dists) != k or len(target_dists) != k + 1:
        raise ValueError("need k draft and k+1 target distributions")
    vocab = len(target_dists[0])
    if vocab == 0:
        raise ValueError("empty vocabulary")
    for dist in draft_dists + target_dists:
        validate_dist(dist, vocab)
    for token, q in zip(draft_tokens, draft_dists):
        if type(token) is not int or not 0 <= token < vocab or q[token] <= 0:
            raise ValueError("draft token must have positive proposal probability")
    accepted: list[int] = []
    for i, token in enumerate(draft_tokens):
        p = target_dists[i]
        q = draft_dists[i]
        if rng.random() < min(1.0, p[token] / q[token]):
            accepted.append(token)
            continue
        return accepted + [residual_sample(p, q, rng)]
    return accepted + [sample(target_dists[k], rng)]

def speculative_speedup(gamma: int, alpha: float, c: float, verify_cost: float = 1.0) -> float:
    if (type(gamma) is not int or gamma < 1
            or not all(math.isfinite(value) for value in (alpha, c, verify_cost))
            or not (0 <= alpha <= 1 and c >= 0 and verify_cost > 0)):
        raise ValueError("invalid parameters for speculative speedup")
    expected_tokens = (1 - alpha ** (gamma + 1)) / (1 - alpha) if alpha < 1.0 else float(gamma + 1)
    cycle_cost = gamma * c + verify_cost
    if not math.isfinite(cycle_cost):
        raise ValueError("cycle cost overflowed")
    return expected_tokens / cycle_cost

rng = random.Random(0)
draft_tokens = [0, 1]
draft_dists = [[0.8, 0.2], [0.3, 0.7]]
target_dists = [[0.7, 0.3], [0.2, 0.8], [0.4, 0.6]]
emitted = speculative_step(draft_tokens, draft_dists, target_dists, rng)

print(f"emitted tokens: {emitted}")
assert emitted == [0, 1, 1]
rejected = speculative_step([0], [[1.0, 0.0]], [[0.0, 1.0], [0.5, 0.5]], random.Random(1))
assert rejected == [1]
assert speculative_step([0], [[1.0, 0.0]], [[1.0, 0.0], [0.0, 1.0]], random.Random(1)) == [0, 1]
try:
    speculative_step([1], [[1.0, 0.0]], [[0.5, 0.5], [0.5, 0.5]], random.Random(1))
except ValueError:
    pass
else:
    raise AssertionError("impossible draft token accepted")
print(f"forced rejection emits: {rejected}")

speedup_fast = speculative_speedup(4, 0.8, 0.1)
speedup_slow = speculative_speedup(4, 0.3, 0.15)
print(f"speedup (gamma=4, alpha=0.8, c=0.1): {speedup_fast:.2f}x")
print(f"speedup (gamma=4, alpha=0.3, c=0.15): {speedup_slow:.2f}x")
assert math.isclose(speedup_fast, 2.401, abs_tol=0.01)
assert speedup_slow < 1.0
print("acceptance, rejection, speedup multipliers, and invalid support checked")
Output
emitted tokens: [0, 1, 1]
forced rejection emits: [1]
speedup (gamma=4, alpha=0.8, c=0.1): 2.40x
speedup (gamma=4, alpha=0.3, c=0.15): 0.89x
acceptance, rejection, speedup multipliers, and invalid support checked

Keep verification cost explicit. Predict what happens to the same acceptance profile if verification takes twice the ordinary target step:

verification-cost-changes-the-win.py
for malformed_gamma, malformed_cost in ((2.5, 0.1), (4, float("inf"))):
    try:
        speculative_speedup(malformed_gamma, 0.8, malformed_cost)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid span/cost accepted")
print(f"verification cost 1T: {speculative_speedup(4, 0.8, 0.1):.2f}x")
print(f"verification cost 2T: {speculative_speedup(4, 0.8, 0.1, verify_cost=2):.2f}x")
print("fractional spans and infinite draft costs rejected")
Output
verification cost 1T: 2.40x
verification cost 2T: 1.40x
fractional spans and infinite draft costs rejected

With this seed both drafts survive, so the first call emits two accepted tokens plus one extra. The second call rejects a proposal with zero target probability and emits its residual correction. Boundary checks aren't a proof of distributional exactness, but they prevent the acceptance-only demo from hiding a broken rejection branch. Follow-on methods such as EAGLE use a lightweight feature-level drafting network informed by target hidden states.[14]

Serving question stays the same: did accepted tokens repay the extra draft and verification work?

Lower precision cuts the bytes in that read

If decode repeatedly streams weights and past KV states, reducing precision is the most direct hardware bandwidth lever. The key is distinguishing which memory stream you are compressing: static model weights or dynamic KV cache buffers.

Many production stacks begin with BF16 or FP16, then adopt FP8 execution when modern accelerators (like NVIDIA Hopper or Ada Lovelace) provide native FP8 Tensor Cores, or use weight-only INT4/INT8 formats (such as AWQ or GPTQ) when the primary goal is shrinking decode memory traffic.[15][16]

Weight quantization: shifting decode toward the compute roof

When weight reads dominate, moving an ideal 8B payload from FP16 (2 bytes/parameter) to an 8-bit format (1 byte) or packed INT4 (0.5 byte) reduces it from 16 GB to 8 GB or 4 GB. Scales, padding, unquantized tensors, and kernel layouts add to those payloads:

  • Serialization: 4 GB/(3,350 GB/s)4\text{ GB}/(3{,}350\text{ GB/s}) is about 1.2 ms versus 4.8 ms for 16 GB. These full-read floors assume peak bandwidth; benchmark decode timings and speedup on the target workload.
  • Intensity: The weight-only formula increases fourfold at the same batch. KV traffic, dequantization, and the compatible compute format change the complete roofline. Compare the B=75B=75 estimate with the target GPU's ridge point, then profile whether Tensor Cores execute the kernel.
  • Capacity: Saving 12 decimal GB releases about 11.18 GiB. If all of it can join the unchanged KV pool, the fixture's 4K payload capacity rises from 103 to 125, adding 22 streams before scales, output growth, or latency limits. That capacity gain is about 22 half-GiB histories.

Keep the batch and histories fixed to isolate traffic. In the earlier full-read model, 64 requests with 4K histories read about 34.36 decimal GB of KV per iteration, already more than the 16 GB weights. Quartering the weights doesn't quarter the combined read. Predict the result when histories grow to 16K:

weight-compression-with-kv-traffic.py
batch = 64
fp16_weights, int4_weights = 16_000_000_000, 4_000_000_000
kv_bytes_per_token = 2 * 32 * 8 * 128 * 2
for history in (4096, 16384):
    kv_read = batch * history * kv_bytes_per_token
    before = fp16_weights + kv_read
    after = int4_weights + kv_read
    # Same achieved bandwidth, full KV reads, unchanged batch and cache width.
    traffic_rate_ratio = before / after
    assert 1 < traffic_rate_ratio < 4
    print(f"history={history}: KV read={kv_read / 1e9:.2f} GB; rate ratio={traffic_rate_ratio:.2f}x")
Output
history=4096: KV read=34.36 GB; rate ratio=1.31x
history=16384: KV read=137.44 GB; rate ratio=1.08x

This yields about 1.31× at 4K and 1.08× at 16K for the assumed bandwidth-only rate, before dequantization or any other overhead. The 64-stream 16K case needs 128 GiB of raw KV, so it exceeds our earlier single-GPU pool; it is a traffic extrapolation for sufficient storage, not a feasible admission setting on that replica. Profile the actual read mix before choosing weight compression, KV compression, or a smaller retained history. Recompute the model if weight savings instead fund a larger batch.

Four stacked byte-read rows share a 0-to-160 decimal-GB scale for batch 64. At 4K, KV stays 34.36 GB while weight reads fall from 16 to 4 GB, changing the assumed total from 50.36 to 38.36 GB and the rate ratio to 1.31x. At 16K, KV stays 137.44 GB and totals fall from 153.44 to 141.44 GB, a 1.08x ratio. The 16K traffic case needs 128 GiB of KV, above the 51.6 GiB pool.
KV stays fixed within each pair as weight reads fall from 16 to 4 decimal GB. At 16K the 64-request KV payload is 128 GiB, above the 51.6 GiB pool, so that row is a traffic extrapolation with sufficient storage, not an admitted single-GPU batch.

KV cache quantization: doubling concurrency and slashing attention bandwidth

KV precision is a separate choice from weight precision, though an execution mode can also quantize queries or other operations. Current vLLM FP8 documentation describes backend-specific schemes, scale calibration, and leaving selected layers unquantized.[17] Formats such as INT8/INT4 require their own supported model/backend path. Compare two ideal payload effects:

  1. Capacity: One byte rather than two halves raw 8B KV from 128 KiB to 64 KiB/token. Flooring prevents exact stream-count doubling: a fixed 51.6 GiB pool's 8K payload capacity goes from 51 to 103, before scale/layout overhead. At 128K it goes from 3 to 6. Admission and SLO-qualified capacity may differ.

  2. Traffic: If each active request's full retained K/V payload is read once from HBM, the model's byte count is:

    Attention_KV_Bytes=2×B×L×Nlayers×Nkv_heads×dhead×Pbytes_kv\text{Attention\_KV\_Bytes} = 2 \times B \times L \times N_{\text{layers}} \times N_{\text{kv\_heads}} \times d_{\text{head}} \times P_{\text{bytes\_kv}}

    Whether it exceeds weight traffic depends on batch, architecture, and history. Halving element width halves this raw term; device bandwidth remains fixed, and measured ITL also reflects cache reuse, repeated loads, scales, conversion, and other work. Evaluate output quality too.

The top tensor stream packs the same 8B parameters into 2B-parameter groups: FP16 occupies 16 decimal GB, INT8 8 GB, and INT4 4 GB before scales or padding. The lower stream holds 8K KV requests in the same 51.6 GiB pool: 16-bit elements fit 51, 8-bit fit 103, and 4-bit fit 206. Cells group eight requests except each final partial group; exact request tails are shown in labels.
The top keeps the weight count fixed while packing each 2B-parameter group at a different width. The bottom fixes weights and pool size while changing KV width. These raw payload floors exclude scales, layouts, backend support, quality, and runtime overhead.
weight-bytes-by-precision.py
import math

parameters = 8_000_000_000
bytes_per_weight = {"FP16": 2.0, "INT8": 1.0, "INT4": 0.5}

for precision, width in bytes_per_weight.items():
    traffic_gb = parameters * width / 1_000_000_000
    print(f"{precision}: {traffic_gb:.1f} GB of weights per full read")

print("INT4 weight traffic is 0.25x FP16 before kernel overhead.")
pool_gib = (80_000_000_000 - 16_000_000_000) / 1024**3 - 8
freed_gib = 12_000_000_000 / 1024**3
old_capacity = math.floor(pool_gib / 0.5)
new_capacity = math.floor((pool_gib + freed_gib) / 0.5)
print(f"12 decimal GB released: {freed_gib:.2f} GiB")
print(f"4K raw KV capacity: {old_capacity} -> {new_capacity} (+{new_capacity - old_capacity})")
assert (old_capacity, new_capacity) == (103, 125)
Output
FP16: 16.0 GB of weights per full read
INT8: 8.0 GB of weights per full read
INT4: 4.0 GB of weights per full read
INT4 weight traffic is 0.25x FP16 before kernel overhead.
12 decimal GB released: 11.18 GiB
4K raw KV capacity: 103 -> 125 (+22)

Common mistake: 4-bit weights aren't primarily an artifact to save disk space. The real serving win is fewer bytes transferred across the memory bus during each decode step. Whether that translates to lower user-facing latency still depends on kernel dequantization overhead and whether model accuracy passes your evaluations.

When one request no longer fits

The 8B replica was a fit-and-batch problem. To stress-test the formula, extrapolate the 70B architecture to 1,000,000 cached tokens: 2×1,000,000×80×8×128×22\times1{,}000{,}000\times80\times8\times128\times2 bytes, about 305 GiB of KV payload. This extrapolation exceeds the documented context length: the original Llama 3 release supports 8K and Llama 3.1 extends it to 128K.[7][5] Sharding memory leaves the checkpoint's trained positional behavior unchanged.

One request has crossed the single-device residency boundary; plan on a shorter context or a sharded serving layout.

splits the sequence dimension across devices, not the layer stack or the batch. Each GPU holds a shard of the prompt and its KV.

Attention still needs information from other shards. Ring Attention is one approach for long-sequence processing: while a device computes blockwise attention on local queries, K/V blocks rotate around the ring.[18] Decode implementations can instead combine partial attention results across cache shards. The communication pattern depends on phase and implementation; don't assume every system circulates all KV on each generated token.

That isn't , which shards work and weights inside a layer, or pipeline parallelism, which shards layers. These techniques can be combined, and tensor parallelism may also shard KV heads.

Llama 3 70B in FP16 is about 140 GB of parameters, so it can't reside on one 80 GB GPU even at batch 1.[5] The next chapter, model parallelism, makes that split its main subject.

context-parallel-kv-shards.py
def kv_gib(sequence_tokens: int) -> float:
    total_bytes = 2 * sequence_tokens * 80 * 8 * 128 * 2
    return total_bytes / 1024**3

total_kv = kv_gib(1_000_000)
devices = 4
print(f"hypothetical 1M-token KV with 70B dimensions: {total_kv:.1f} GiB")
print(f"evenly sharded over {devices} devices: {total_kv / devices:.1f} GiB/device")
print("KV only: weights, communication buffers, and runtime need additional memory.")
Output
hypothetical 1M-token KV with 70B dimensions: 305.2 GiB
evenly sharded over 4 devices: 76.3 GiB/device
KV only: weights, communication buffers, and runtime need additional memory.
A hypothetical one-million-token request with Llama 3 70B dimensions has 305.2 GiB of raw KV, or 76.3 GiB on each of four equal sequence shards; each device also needs unquantified weights and buffers. During one prefill Ring Attention approach, K/V blocks rotate between sequence shards. During decode, shard-local partial attention states combine into one result; the figure does not imply every decode circulates all KV. The lower axis diagrams distinguish context parallelism across tokens, tensor parallelism within a layer, and pipeline parallelism across layer depth. Sharding does not extend supported context.
The 1M-token example extrapolates 70B FP16 geometry beyond supported context. Ring Attention is one prefill communication pattern; decode can combine shard-local partial results. Neither layout removes weights, buffers, communication, or the checkpoint context limit.

When the budget lies

These traces show up when formulas look fine on paper but users or profiles disagree. Start with the symptom, name the ceiling it points to, then test one targeted fix.

"I bought a higher-FLOP GPU and decode barely moved"

Peak compute went up, but tokens per second didn't. That observation alone doesn't identify the bottleneck. Profile HBM, kernels, transfers, and host scheduling. If full reads saturate bandwidth while compute has headroom, test amortization or fewer bytes before expecting extra peak FLOPs to help.

"Throughput is up and chat feels worse"

Aggregate TPS rose after you lifted max batch, but users report sluggish streams. Inspect iteration duration, queue wait, and delivery by request class; mean TPOT can hide gaps. Gate on the product's TTFT and ITL targets, not replica tokens/s alone. Separate traffic classes if their interference is the measured problem.

"Speculative decoding added latency"

A draft model is in the path, and p50 decode got worse. Acceptance may be too low, or draft-plus-verify overhead may exceed the saved target passes. Measure accepted tokens per draft span before . If most proposals die early, keep ordinary decode.

"HBM climbs even though every session shares a system prompt"

KV usage looks close to B×LpromptB\times L_{\text{prompt}} despite repeated text. Copy-on-write isolates writes after sharing is established; it doesn't discover matching prefixes. Cross-request prefix caching needs a compatible available entry and trusted sharing scope. Branching inside one request can share directly. Inspect allocated pool bytes separately from live/cached block ownership, then confirm actual reuse and reference counts.

Size the 8B replica

Put the pieces together with an explicit accounting budget. This simplified partition includes reserved/unavailable space in the last term; real engines have more detailed categories:

Mtotal=Mweights+MKV_pool+Mactivations+MslackM_{\text{total}} = M_{\text{weights}} + M_{\text{KV\_pool}} + M_{\text{activations}} + M_{\text{slack}}

For this fixture, account for:

  1. Weights: Approximate the 8B-class FP16 payload as 16.0×10916.0\times10^9 bytes, or 14.90 GiB. The exact checkpoint, packing, adapters, and execution path determine its real allocation.
  2. KV pool: Physical storage from which requests use blocks. A preallocated pool can keep the same device allocation while live block ownership grows or shrinks.
  3. Activations/workspace: Intermediate tensors, kernel scratch, and captured-graph buffers depend on model, batch/chunk shapes, dtype, and backend. FlashAttention avoids the full score/probability intermediates; it doesn't remove all workspace. Measure the peak rather than assume 2–4 GiB universally.
  4. Other allocations/reserve: Runtime allocations, allocator slack, other GPU users, and intentionally unused headroom. Host scheduling queues aren't automatically VRAM allocations. vLLM's gpu_memory_utilization limits the model executor's fraction, rather than specifying a fixed activation/slack split; explicit kv_cache_memory_bytes overrides automatic pool sizing.[19]

GB means 10910^9 bytes; GiB means 2302^{30} bytes. Assume exactly 80,000,000,000 budget bytes here, or 74.51 GiB. Don't infer the runtime's available byte count from a nominal “80 GB” product label; inspect device-reported total/free memory and engine allocations. Subtract the approximate weights and an assumed 8 GiB combined workspace/other reserve:

MKV_pool=74.51−14.90−8.00=51.60 GiBM_{\text{KV\_pool}} = 74.51 - 14.90 - 8.00 = \mathbf{51.60\text{ GiB}}

Now compute single-request payload for a 4,096-token sequence on Llama 3 8B:

KV bytes=2×1×4096×32×8×128×2=536,870,912 bytes=0.50 GiB\text{KV bytes} = 2\times1\times4096\times32\times8\times128\times2 = 536{,}870{,}912\text{ bytes} = 0.50\text{ GiB}

The theoretical payload ceiling is:

Bmax=⌊51.60 GiB0.50 GiB⌋=103 concurrent requestsB_{\text{max}} = \left\lfloor \frac{51.60\text{ GiB}}{0.50\text{ GiB}} \right\rfloor = \mathbf{103\text{ concurrent requests}}

That snapshot counts 4,096 retained positions per request with no future growth. Reserve 512 more positions, giving 4,608 positions or 0.5625 GiB per request, and this fixture's growth-budget ceiling drops to:

Bmax_safe=⌊51.60 GiB0.5625 GiB⌋=91 concurrent requestsB_{\text{max\_safe}} = \left\lfloor \frac{51.60\text{ GiB}}{0.5625\text{ GiB}} \right\rfloor = \mathbf{91\text{ concurrent requests}}

Average context lengths aren't admission guarantees: a sudden burst of long prompts can exhaust physical blocks even if the historical average was 4K.

vram-capacity-headroom.py
import math

kv_bytes = 2 * 1 * 4_096 * 32 * 8 * 128 * 2
kv_per_request_gib = kv_bytes / 1024**3
device_gib = 80_000_000_000 / 1024**3
weight_gib = 16_000_000_000 / 1024**3
runtime_gib = 8
kv_budget_gib = device_gib - weight_gib - runtime_gib
arithmetic_ceiling = math.floor(kv_budget_gib / kv_per_request_gib)
growth_gib = (4_096 + 512) * (2 * 32 * 8 * 128 * 2) / 1024**3
growth_ceiling = math.floor(kv_budget_gib / growth_gib)

print(f"KV per 4K request: {kv_per_request_gib:.2f} GiB")
print(f"KV budget after weights and runtime: {kv_budget_gib:.2f} GiB")
print(f"arithmetic-only batch ceiling: {arithmetic_ceiling}")
print(f"ceiling with 512 output tokens reserved: {growth_ceiling}")
assert arithmetic_ceiling == 103 and growth_ceiling == 91
assert 91 * growth_gib <= kv_budget_gib < 92 * growth_gib
Output
KV per 4K request: 0.50 GiB
KV budget after weights and runtime: 51.60 GiB
arithmetic-only batch ceiling: 103
ceiling with 512 output tokens reserved: 91
An assumed 80,000,000,000-byte device budget is 74.5058 GiB. A proportional partition subtracts 14.9012 GiB of 8B FP16 weights and an assumed 8 GiB runtime reserve, leaving a 51.6046 GiB KV pool. On one shared pool scale, 103 current 4K payloads use 51.5 GiB; reserving 512 output tokens leaves room for 91 requests at 0.5625 GiB each. A 92nd requires 51.75 GiB and exceeds the pool by 0.1454 GiB.
The total, weight estimate, and 8 GiB reserve are assumptions, not a device measurement. Counts show payload-only ceilings; page rounding, future growth, latency, and workload tails affect admission.

Try a page-aware admission calculation next. Use 16-token blocks, count each request's prompt plus reserved output, and sum the rounded allocations. For prompts [4090, 4096, 4100] with 512 reserved output tokens each, predict whether three times the average length gives the exact allocation.

page-aware-admission.py
BYTES_PER_TOKEN = 2 * 32 * 8 * 128 * 2
BLOCK_TOKENS = 16

def reserved_bytes(prompt_tokens: int, output_tokens: int) -> int:
    if (type(prompt_tokens) is not int or type(output_tokens) is not int
            or prompt_tokens <= 0 or output_tokens < 0):
        raise ValueError("use integer counts: prompt positive, output reserve nonnegative")
    blocks = (prompt_tokens + output_tokens + BLOCK_TOKENS - 1) // BLOCK_TOKENS
    return blocks * BLOCK_TOKENS * BYTES_PER_TOKEN

prompts = [4090, 4096, 4100]
allocations = [reserved_bytes(prompt, 512) for prompt in prompts]
payload = sum(prompt + 512 for prompt in prompts) * BYTES_PER_TOKEN
reserved = sum(allocations)
assert reserved >= payload
assert reserved - payload < len(prompts) * BLOCK_TOKENS * BYTES_PER_TOKEN
assert reserved_bytes(4096, 512) < reserved_bytes(4097, 512)
print("reserved token slots:", [value // BYTES_PER_TOKEN for value in allocations])
print("unused rounded slots:", (reserved - payload) // BYTES_PER_TOKEN)

def admit(current_bytes: int, new_request_bytes: int, budget_bytes: int) -> bool:
    return current_bytes + new_request_bytes <= budget_bytes

exact_fit = admit(sum(allocations[:2]), allocations[2], reserved)
too_small = admit(sum(allocations[:2]), allocations[2], reserved - 1)
assert exact_fit and not too_small
print("last request fits exact budget:", exact_fit)
print("last request fits one byte less:", too_small)
Output
reserved token slots: [4608, 4608, 4624]
unused rounded slots: 18
last request fits exact budget: True
last request fits one byte less: False
A zoom into each request’s final 16-token KV block shows prompt plus 512 reserved output positions. A 4,090-token prompt needs 4,602 positions, so 4,608 are allocated and six slots remain unused. A 4,096-token prompt lands exactly on 4,608. A 4,100-token prompt needs 4,612, so 4,624 are allocated and twelve slots remain unused. The three reservations total 13,840 slots with 18 unused; a fixture budget equal to their summed bytes fits, while one byte less rejects the third request.
The figure magnifies each final 16-token block; earlier blocks are omitted. Rounding is per request, with 18 unused slots total. The exact-fit budget is defined by these three reservations and is separate from the full 51.60 GiB pool.

The answer is 4,608, 4,608, and 4,624 slots: 18 unused slots in total. The last prompt crosses a page boundary. This calculation deliberately excludes shared prefixes and cache eviction. It models a conservative reservation for these requests, not a complete allocator. On a serving engine, verify peak allocated memory, preemptions, TTFT, and token-gap percentiles under mixed-length load before turning such a calculation into an admission limit.

Check whether a replica is waiting on bandwidth or on KV residency. Turn the 80 GB label into a batch ceiling that still has room for the answer to grow.

Complete the lesson

Mastery Check

Answer every question, then check your score. Score 75% or higher to mark this lesson complete.

1.A full-attention toy stores K and V at 24 layers, 4 KV heads, width 64 and 2 bytes/value. Each request has 6000 prompt positions plus 1000 reserved output positions, rounded independently to 16-position pages. A 3 GiB pool has no other overhead. How many equal reservations fit?
2.An authored decode step has 8 active requests. It reads 12 decimal GB of weights and 4 decimal GB of total current KV once, at an achieved 500 GB/s. Ignore other work. What serialization time and aggregate output rate follow?
3.Two deployments each cost $4.80 per hour over their complete billed intervals. One sustains 800 useful output tokens/s, the other 400. What are their GPU-only costs per million useful tokens?
4.Four already-prefilled requests have remaining lengths [1, 4, 2, 2] and two slots. A static policy finishes each consecutive pair before starting the next. A continuous policy refills a vacated slot before the next iteration. Each live request emits one token per iteration. How many idealized weight reads do the policies need?
5.Requests A and B map to the same partially filled four-slot page 9 containing [u, v, empty, empty], with reference count 2. They will append different private tokens. The shared prefix is already found. Which state transition preserves both histories?
6.In a scheduling toy, 4 live decode requests each take one new-position slot from a 10-position mixed-step budget; one prompt still has 20 positions. Separately, a phase split transfers 1.5 GiB at 100 GiB/s, adds 4 ms coordination and avoids 18 ms interference, with no overlap. What first prompt chunk and net latency change follow?
7.A serial speculative model uses gamma 3, iid acceptance mean 0.5, draft-step cost 0.2 baseline steps and verification-plus-overhead 1.2 steps. Assume long outputs. What expected tokens per cycle and rate ratio follow?
8.For one speculative proposal, p=[0.1, 0.6, 0.3], q=[0.5, 0.25, 0.25], the draft token is ID 0, and the supplied uniform is 0.4. Later draft tokens are pending. Which rejection decision and correction distribution over IDs 0/1/2 are correct?
9.A fixed-batch full-read toy moves 10 GB of weights and 30 GB of KV each step at unchanged achieved bandwidth. Compare two separate changes: quarter only the weights, or halve only the KV. What bandwidth-only rate ratios follow, before conversion or kernel overhead?
10.A fresh authored replica has 32 GiB total, 8 GiB weights and 4 GiB runtime reserve. After a cache-width change, each current history is 0.25 GiB and each output-reserved allocation 0.3125 GiB. Read weights plus current histories once at 400 GiB/s. Among batches 16/32/48/64, choose the largest fitting the pool and a 40 ms iteration limit. What choice and modeled aggregate rate follow?

10 questions remaining.

References

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.

Zhong, Y., et al. · 2024 · OSDI 2024

https://arxiv.org/abs/2401.09670

Roofline: An Insightful Visual Performance Model for Multicore Architectures

Williams, S., Waterman, A., & Patterson, D. · 2009

https://doi.org/10.1145/1498765.1498785

Efficiently Scaling Transformer Inference

Pope, R., et al. · 2023 · MLSys 2023

https://proceedings.mlsys.org/paper_files/paper/2023/hash/c4be71ab8d24cdfb45e3d06dbfca2780-Abstract-mlsys2023.html

H100 GPU

NVIDIA · 2026

https://www.nvidia.com/en-us/data-center/h100/

The Llama 3 Herd of Models.

Dubey, A., et al. · 2024 · arXiv preprint

https://arxiv.org/abs/2407.21783

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.

Ainslie, J., et al. · 2023 · EMNLP 2023

https://arxiv.org/abs/2305.13245

Meta Llama 3 Model Card

Meta · 2024

https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

Efficient Memory Management for Large Language Model Serving with PagedAttention.

Kwon, W., et al. · 2023 · SOSP 2023

https://arxiv.org/abs/2309.06180

Orca: A Distributed Serving System for Transformer-Based Generative Models.

Yu, G.-I., et al. · 2022 · OSDI 2022

https://www.usenix.org/conference/osdi22/presentation/yu

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

Agrawal, A., et al. · 2023 · arXiv preprint

https://arxiv.org/abs/2308.16369

Optimization and Tuning.

vLLM · 2026

https://docs.vllm.ai/en/latest/configuration/optimization.html

Splitwise: Efficient Generative LLM Inference Using Phase Splitting.

Patel, P., et al. · 2023

https://arxiv.org/abs/2311.18677

Fast Inference from Transformers via Speculative Decoding.

Leviathan, Y., Kalman, M., & Matias, Y. · 2023 · ICML 2023

https://arxiv.org/abs/2211.17192

EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.

Li, Y., et al. · 2024 · ICML 2024

https://arxiv.org/abs/2401.15077

FP8 Formats for Deep Learning.

Micikevicius, P., et al. · 2022

https://arxiv.org/abs/2209.05433

GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers

Frantar, E., et al. · 2023 · ICLR 2023

https://arxiv.org/abs/2210.17323

Quantized KV Cache

vLLM Team · 2026 · vLLM Documentation

https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/

Ring Attention with Blockwise Transformers for Near-Infinite Context.

Liu, H., et al. · 2023 · arXiv preprint

https://arxiv.org/abs/2310.01889

vLLM Cache Configuration

vLLM Project · 2026

https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.