Scaling LLM Inference

Continuous batching can refill decode slots as soon as a request finishes. A full queue can still hide a capacity failure: once the scheduler is willing to admit more work, what runs out first on the GPU serving a large language model (LLM)?

Picture a code-assistant pool during a release freeze. Every answer still arrives one token at a time, while short completions and long patch explanations share one replica.

Each decode step rereads model weights from GPU high-bandwidth memory (HBM) and consults a growing KV cache. Those bytes, rather than peak FLOPs, usually decide how many streams fit and what each token costs.

The scheduler can refill slots, but a replica still has finite budget. We will follow one mixed-length assistant: an 8B-class replica that fits on one 80 GB GPU, a 70B-class KV cache that overruns that card, and a throughput-latency-cost choice that works for both.

Why isn't "keep the batch full" enough to scale LLM serving?

Answer

A full batch still has to fit in HBM and still has to move weights plus KV state every decode step. If bandwidth or KV residency is the ceiling, extra admitted requests queue or stall instead of raising useful tokens per second.

Decode sits under a bandwidth roof

You already know the two phases from inference mechanics. Prefill reads known prompt tokens in parallel and is often compute-heavy.

Decode emits one new token per request step. This dependency is why a request can have plenty of prompt work behind it but still need many sequential steps to finish.

Users feel prefill as time to first token (TTFT). They feel decode as inter-token latency (ITL), also called time between tokens (TBT). DistServe defines time per output token (TPOT) as average time to generate each output token after first, so mean TPOT can look fine while p95 ITL is ugly.^{[1]Reference 1DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

The scaling fact is the intensity split. Arithmetic intensity is FLOPs per byte loaded from HBM.

Prefill reuses each weight matrix across many prompt positions. Small-batch decode reuses those same weights for one new position per request, so each step looks more like a thin matrix-vector read.

The roofline model names the resulting ceiling: attainable work is the minimum of peak compute and memory bandwidth times intensity. Below the ridge, bandwidth is the limit. Above it, compute is the limit.^{[2]Reference 2Roofline: An Insightful Visual Performance Model for Multicore Architectureshttps://doi.org/10.1145/1498765.1498785[3]Reference 3Efficiently Scaling Transformer Inference.https://arxiv.org/abs/2211.05102}

Before doing hardware arithmetic, predict where each phase lands. One-token decode should sit on the rising bandwidth slope; a large batch or long prefill should move right, toward the flat compute roof.

Name hardware before doing arithmetic. We use NVIDIA H100 SXM as a reference point: it has 80 GB of HBM and peak HBM bandwidth of 3.35 TB/s (3,350 GB/s in decimal units).^{[4]Reference 4H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/}

Llama 3 8B is an 8B-parameter dense model.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} An FP16 copy of those weights is about 16 GB (8e9 * 2 bytes).

If one uncached decode step streamed that whole footprint for one active token, the weight-read upper bound on H100 SXM would be 3350 / 16 ≈ 209 steps per second. That is a bandwidth thought experiment, not a promise about a runtime.

Real kernels also read KV state, miss peak bandwidth, and pay runtime overhead, so observed tokens per second are lower. Use the bound as a diagnosis: if a trace saturates HBM while tensor cores sit idle, buying more FLOPs will not fix decode.

You profile decode and see tensor cores idle while HBM bandwidth is near saturation. Should you first buy more FLOPs or reduce memory traffic?

Answer

Reduce memory traffic first. The profile says the bottleneck is bandwidth, so extra FLOPs will sit idle too. Better first moves are a larger effective decode batch, weight quantization, GQA/MQA, KV compression, or better cache packing.

KV bytes cap the batch you can form

Batching only helps if extra requests fit. The KV cache lesson derived the payload formula: each token stores a Key and a Value at every layer and KV head.

Llama 3 uses grouped-query attention with 8 KV heads at 8B, 70B, and 405B.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783[6]Reference 6GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.https://arxiv.org/abs/2305.13245} That head count is why KV can be much smaller than a full set of attention heads, but it still grows with layers, context, and concurrency.

For Llama 3 70B in FP16, one token is 2 * 80 * 8 * 128 * 2 = 327,680 bytes, or 320 KiB. Do a rough check before reading on: multiplying by 8,192 should land near a few GiB.

The exact estimate is 2.5 GiB per 8,192-token request. Sixty-four concurrent 8K requests need 160 GiB of KV alone, before weights. An 80 GB H100 can't hold that batch, so a plan to keep raising batch size until the roofline moves hits residency first.

The 8B replica in the same family is the one that fits. Llama 3 8B has 32 layers, 8 KV heads, and d_head = 128, so one token is 128 KiB and a 4K request is 0.5 GiB.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

That is the budget we will size later. The 70B numbers show the same formula crossing a GPU before FLOPs do.

That 160 GiB figure assumes every request occupies a live 8K payload. A max-context slab reservation is worse: it pays for tokens that never arrive.

PagedAttention allocates fixed-size blocks on demand and maps logical token order through a block table, so finished requests return pages to a shared pool.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} Paging doesn't shrink bytes per live token. It stops the allocator from pinning unused tail slots.

In the KV-cache formula, which terms can the serving stack change without shortening user context?

Answer

Grouped-query or multi-query attention reduces num_kv_heads. KV-cache quantization reduces dtype_bytes. Batch size is a workload and admission choice. Layer count and head dimension stay fixed for a chosen checkpoint.

The throughput, latency, cost triangle

Once bandwidth and KV residency are in view, "maximize tokens per second" is not a complete objective. Start with product promise: interactive users care about TTFT and p95 ITL, while offline jobs care more about completion rate.

Larger batches amortize weight reads and usually raise aggregate tokens per second (TPS). They also make requests compete for HBM, which can stretch TTFT and p95 ITL.

Cost per token follows sustained throughput, not the sticker GPU price. A slower, latency-protected replica can therefore be more expensive per token and still be correct for its workload.

Cost per token is hourly GPU spend divided by tokens actually produced:

$\text{cost per token} = \frac{\text{GPU \$ per hour}}{\text{sustained tokens per second} \times 3600}$

Treat $3.00/hour as a worked fixture for August 2026, not a live cloud quote. Suppose a well-batched replica sustains 2,500 decode tokens/s across active requests:

• Tokens per hour = 2,500 * 3,600 = 9,000,000
• Cost per million tokens = $3.00 / 9,000,000 * 1,000,000 = $0.33

Starve the batch to 250 tokens/s on the same GPU-hour and cost per million jumps to $3.33. Utilization is a 10x multiplier. Batching is not only a latency knob; it moves the cost corner of the triangle.

The triangle has a blunt rule: push two corners hard and the third drifts. Cap batch size for a 500 ms TTFT chat SLO, and cost per token climbs because the GPU is underused.

Push batch size for overnight summarization, and tail latency rises because no human is waiting on each token. There is no universal best point, only one that meets the product latency SLO at acceptable cost.

Production tip: Pick the operating point from the product SLO, then size hardware to it. Watch GPU KV usage, prefill backlog, and decode queue depth together. High KV usage plus rising TTFT usually means residency is capping concurrency. Low KV usage with idle compute means you're leaving throughput, and cheap tokens, on the table.

Two teams serve the same 8B model on the same GPU SKU but report a 5x gap in cost per million tokens. The interactive team is more expensive. Why isn't that automatically a misconfiguration?

Answer

Interactive serving caps batch size to protect TTFT and ITL, so each GPU-hour spreads over fewer tokens. The batch team pushes until throughput saturates. Both can be correct operating points for their SLOs.

Batching raises arithmetic intensity

The previous chapter treated continuous batching as slot reuse: when request A finishes, D can enter at the next decode iteration instead of waiting for C's long answer.^{[8]Reference 8Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu} Slot reuse is only half the story. One decode step still streams the weight tensor roughly once for the whole batch. More active tokens on that step means more useful FLOPs per byte, which is how serving climbs the roofline without changing the model.

Static batching fights that climb. If three assistant requests need 2, 2, and 5 remaining tokens, a request-level batch of size 3 spends later steps on padding or idle slots. Iteration-level scheduling keeps the weight read attached to live work.

Predict the comparison before running it: same weight-read steps should produce more tokens when finished slots refill immediately. This sketch counts weight-read steps against emitted tokens. It isn't a production scheduler: no token budget, no prefill chunking, no KV admission check. It isolates why membership changes raise work per read.

Same five weight-read steps, more tokens, because D and E filled slots that static batching would have padded. The intensity gain still stops where KV residency stops. A scheduler that admits past the cache budget just swaps bandwidth headroom for eviction and preemption.

Why does batching help decode even though every request still needs its own next token?

Answer

The model weights can be read once for a larger set of active tokens, so that memory movement is amortized across requests. Batching doesn't remove the sequential dependency inside each request. It raises useful work per weight read until KV memory or latency SLOs say stop.

Sharing, chunking, and phase isolation

Three more levers change the same budget without training a new model. Each targets a different source of pressure: repeated prompt work, prompt interference, or movement between phase-specific workers.

Prefix sharing. Many assistant requests start from the same system prompt. Prefix caching is the lookup that finds reuse across independent requests.

Copy-on-write is what happens after sharing exists: continuations keep the same physical blocks until one has to write, then runtime clones only the dirty block.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} For Llama 3 8B, a shared 4K system prompt is 0.5 GiB the first time and nearly free for each later session that hits the cache, until someone diverges.

Chunked prefill. A 10K RAG prompt on the same worker as live streams can stall ITL. Sarathi-Serve and current vLLM V1 scheduling bound that interference by giving leftover max_num_batched_tokens to prefill after decode and splitting a prompt that doesn't fit.^{[9]Reference 9Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.https://arxiv.org/abs/2308.16369[10]Reference 10Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

Chunking doesn't add HBM. It trades TTFT on a long prompt for smoother ITL on streams already admitted. Details live in the scheduling chapter.

Prefill-decode disaggregation. Splitwise and DistServe move the two phases onto separate pools when avoided interference pays for KV transfer.^{[11]Reference 11Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[1]Reference 1DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670} This changes worker placement, not payload size.

For a Llama 3 70B 8K request, payload is 2.5 GiB. Over a 200 GiB/s link, the one-way floor is 12.5 ms before protocol and scheduling. That is a different break-even from the smaller Qwen handoff in the scheduling chapter: long 70B prefixes make transfer expensive, so disaggregation must save more queueing than 12.5 ms plus overhead.

Bandwidth sets how hard you can push the batch. KV residency sets how many requests that batch can contain. The operating point is where those two ceilings meet the product's latency SLO.

When can prefill-decode disaggregation make latency worse?

Answer

When KV-cache transfer, coordination, or extra scheduling cost more than the head-of-line blocking they remove. Short prompts, light traffic, or a slow interconnect often favor a colocated worker.

Speculative decoding buys tokens per weight read

Batching amortizes one target-model weight read across requests. Speculative decoding applies the same idea within one request: one target read can score several future tokens.

A small draft model proposes k tokens. The large target model scores that span in one forward pass. Accepted prefix tokens plus a correction can replace several ordinary target decode steps.^{[12]Reference 12Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192} The dedicated speculative-decoding lesson goes deeper; capacity reading here is simple: fewer target-weight streams per emitted token, if the draft is right often enough.

The accept/reject rule from Leviathan et al. can keep the target model's distribution. On a mismatch it resamples from the residual max(p - q, 0).

If all k drafts survive, it samples one extra token from the target distribution at the last draft position. Exactness doesn't imply speedup. Low acceptance, an expensive draft, or a clumsy kernel can lose to ordinary decoding.

This stdlib step uses a two-token vocabulary so you can check the residual path without a model runtime. p is the target distribution, q is the draft distribution, and the draft proposed token 0 then token 1.

With this seed both drafts survive, so the step emits two accepted tokens plus one extra from the target. Follow-on methods such as EAGLE draft from the target model's hidden states instead of a separately trained small model.^{[13]Reference 13EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.https://arxiv.org/abs/2401.15077}

Serving question stays the same: did accepted tokens repay the extra draft and verification work?

Why can speculative decoding be exact yet still slower than normal decoding?

Answer

The accept/reject rule can preserve the target distribution, but runtime depends on acceptance and overhead. If the draft is wrong too often, or verification is expensive, saved target passes don't cover the extra work.

Lower precision cuts the bytes in that read

If decode rereads weights, fewer bytes per weight is a direct bandwidth lever. Start by asking whether the profile points to weight traffic or KV traffic.

Many stacks start from BF16/FP16, then move to FP8 when hardware and kernels exist, or to INT8/INT4-style weight-only quantization when the goal is a thinner decode stream.^{[14]Reference 14FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433[15]Reference 15GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformershttps://arxiv.org/abs/2210.17323}

KV-cache quantization is a different knob: it targets residency and long-context batch size once the cache, not the weights, is the cap.^{[16]Reference 16SnapKV: Compressing KV Cache by Selecting Global Attention Patterns.https://arxiv.org/abs/2403.19925[17]Reference 17Quantized KV Cachehttps://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/}

On the 8B replica, an FP16 full-weight read is 16 GB. INT4 is 4 GB of weight payload before scales and overhead.

That can move the single-stream ceiling, but only if the kernel doesn't spend savings on dequantization and quality still clears your eval. The next lesson, model quantization, is where those codecs live.

Common mistake: 4-bit weights aren't mainly a disk-size trick. The serving win is fewer bytes reread during decode. Whether that becomes a latency win still depends on kernels, dequant overhead, and quality.

Why does weight-only INT4 often help decode latency more directly than prefill latency?

Answer

Decode repeatedly streams model weights for small per-token work, so fewer bytes per weight cut bandwidth pressure. Prefill uses larger matrix operations and can be compute-heavy, so weight compression isn't always the dominant lever there.

When one request no longer fits

The 8B replica was a fit-and-batch problem. Some traffic is not. Before choosing hardware, test long-context demand: a 1,000,000-token prompt on Llama 3 70B needs 2 * 1_000_000 * 80 * 8 * 128 * 2 bytes of KV, about 305 GiB.

That isn't a "buy one more H100" situation. One request has crossed the single-device residency boundary.

Context parallelism (CP) splits the sequence dimension across devices, not the layer stack or the batch. Each GPU holds a shard of the prompt and its KV.

Attention still needs cross-shard Keys and Values, so implementations use ring-style exchange: while a device computes blockwise attention on local queries, K/V blocks rotate around the ring.^{[18]Reference 18Ring Attention with Blockwise Transformers for Near-Infinite Context.https://arxiv.org/abs/2310.01889} Context length can scale roughly with device count until communication becomes the next roof.

That isn't tensor parallelism, which shards weights inside a layer, or pipeline parallelism, which shards layers. Those appear when weights don't fit.

Llama 3 70B in FP16 is about 140 GB of parameters, so it can't reside on one 80 GB GPU even at batch 1.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} The next chapter, model parallelism, makes that split its main subject.

What bottleneck does context parallelism target, and what bottleneck can it introduce?

Answer

It targets per-request context and KV that no longer fit on one GPU. It can introduce communication overhead because devices must exchange partial attention state across sequence shards.

When the budget lies

These traces show up when formulas look fine on paper but users or profiles disagree. Start with the symptom, name the ceiling it points to, then test one targeted fix.

"I bought a higher-FLOP GPU and decode barely moved"

Peak compute went up, but tokens per second didn't. Decode was already bandwidth-bound, so extra tensor cores wait on the same HBM reads. Profile HBM first. If it's saturated, raise effective batch, quantize weights, shrink KV, or choose a higher-bandwidth SKU.

"Throughput is up and chat feels worse"

Aggregate TPS rose after you lifted max batch, but users report sluggish streams. You optimized the cost corner and spent the latency corner; mean TPOT can still look acceptable. Gate on p95 ITL/TBT and TTFT, not replica tokens/s alone. Split interactive traffic from offline batch.

"Speculative decoding added latency"

A draft model is in the path, and p50 decode got worse. Acceptance may be too low, or draft-plus-verify overhead may exceed the saved target passes. Measure accepted tokens per draft span before rollout. If most proposals die early, keep ordinary decode.

"HBM climbs even though every session shares a system prompt"

KV usage looks close to batch * full_prompt despite a shared prefix. Copy-on-write preserves blocks only after a prefix-cache hit. Without a lookup and a tenant isolation policy, each request builds its own prompt KV. Enable prefix caching with an explicit sharing policy, then confirm block reference counts in the allocator.

A dashboard shows high TTFT, a rising prefill backlog, stable TPOT, and moderate KV memory. Which subsystem should you inspect first?

Answer

Inspect prefill scheduling and admission first. Stable TPOT suggests decode isn't currently degrading, while high TTFT plus a prefill backlog points to prompt work or queueing before the first token.

Size the 8B replica

Put pieces on one GPU. We want to serve Llama 3 8B FP16 on a single 80 GiB H100: about 16 GiB for weights, 8 GiB reserved for activations, allocator slack, and runtime, and average assistant context of 4,096 tokens. Before calculating, predict whether arithmetic-only KV ceiling is dozens or hundreds of requests.

One 4K request works out as:

KV budget = 80 - 16 - 8 = 56 GiB

Maximum batch = 56 / 0.5 = 112 concurrent 4K requests

That's a payload ceiling, not a production setting. Long-context bursts, fragmentation, CUDA graphs, and latency SLOs all eat headroom. A serving engineer caps well below 112 and watches measured HBM plus p95 ITL.

Why isn't that 112-request ceiling a safe production max concurrency?

Answer

It ignores fragmentation, temporary tensors, prefix-cache behavior, traffic with longer contexts, and latency SLOs. Production capacity should come from measured headroom, not from a closed-form KV quotient alone.