Scaling LLM Inference

Continuous batching keeps decode slots useful. The capacity question for scaling large language model (LLM) inference is harder: when requests, model weights, and KV state all compete for HBM, what limits serving concurrency?

A code assistant that answers "why is this test failing?" still generates every response one token at a time. It's not because the model is "thinking." During decode, the serving stack keeps rereading billions of model weights from GPU memory and consulting a growing KV cache, and that memory movement takes time. Reading the prompt happens in parallel; streaming the answer repeatedly revisits the same large weights and cache state.

Tie together prefill, decode, batching, KV-cache memory, PagedAttention, disaggregation, speculative decoding, and quantization so serving bottlenecks become measurable instead of mysterious. The thread running through all of them is one decision: where to sit on the throughput, latency, and cost triangle for your workload. For the scheduling loop behind this capacity model, see continuous batching.

The two phases of generation

LLM inference is distinct from training because it consists of two very different computational phases: Prefill and Decode. Understanding this distinction is the first step to optimization.

A code-assistant request makes the two phases visible. When a user sends "Why did auth.test.ts fail after this refactor?", the system first has to read the entire prompt and attached context. That's the prefill phase. Then it starts answering, generating one token at a time: "The", "mock", "still", "uses", "the", "old", "schema." That's the decode phase.

Prefill: reading the prompt in one go

In the prefill phase, the model processes the entire user prompt in parallel. This is similar to training: the GPU receives a matrix of shape [batch_size, prompt_len, hidden_dim] and computes attention for all tokens simultaneously.

Because all the input tokens are known upfront, the attention mechanism can compute the interactions between every token in the prompt at once. This parallel processing allows the GPU to use its massive matrix multiplication engines efficiently. A long prefill usually dominates Time To First Token (TTFT), though TTFT also includes queueing and scheduling delay before the first output token is emitted. This visual shows how all tokens in the prompt are processed simultaneously to generate the first output token.

Key characteristics

• Often compute-heavy: Prefill exposes large matrix multiplications. FlashAttention keeps attention exact while reducing HBM traffic relative to materializing the full attention matrix; it doesn't make all attention IO linear in sequence length.^{[1]Reference 1FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.https://arxiv.org/abs/2205.14135}
• Parallel-friendly: Processing many prompt positions together can drive much higher tensor-core utilization than one-token decode. Whether it saturates compute depends on sequence shape, kernel, and hardware.
• Latency: Time usually grows with prompt length, and long prompts often dominate TTFT.

Decode: answering one word at a time

Once the first token is generated, the model switches to autoregressive generation. It generates one token at a time, feeding it back as input for the next step.

Unlike the prefill phase, decoding can't be parallelized across tokens inside one request because each new token depends on the previous ones. The system is locked into a sequential, step-by-step loop. Users feel this as inter-token latency (ITL), also called time between tokens (TBT): the delay between visible streamed tokens. Time Per Output Token (TPOT) is closely related but isn't identical. DistServe, for example, defines TPOT as the average time to generate each output token after the first.^{[2]Reference 2DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

A rate such as tokens per second (TPS) is another view of decode speed. Be careful about the denominator: one stream's TPS isn't the same metric as aggregate tokens per second across a whole server. TTFT affects initial responsiveness, while ITL/TBT and TPOT describe the pace after streaming starts. During autoregressive decoding, each generated token becomes input for the next step.

Key characteristics

• Often memory-bound: A decode step needs model weights and the KV state used by attention. At small or latency-sensitive batches, repeated reads commonly make HBM bandwidth the ceiling; batching can raise arithmetic intensity by sharing weight reads across active requests.
• Low arithmetic intensity at small batches: The arithmetic intensity (FLOPs/byte, i.e., Floating Point Operations per byte of data loaded) can be low because the runtime moves large tensors for only one new position per sequence.

Why decode is memory-bound

Decode-heavy LLM serving is often memory-bandwidth bound, not compute-bound. In a compute-bound operation, the system is bottlenecked by the mathematical calculations it must perform. Training and long-prefill workloads typically expose much larger matrix operations than interactive decode, so they can drive compute hardware more effectively.

During token generation, the bottleneck often shifts toward memory movement. Each new token needs model weights and attention state, but contributes only one new position per active request. At modest decode batches, this produces low arithmetic intensity and makes HBM traffic a central constraint.

A 7 billion parameter model stored in 16-bit precision has weights of about 14 GB in decimal units. If one uncached decode step had to read that full weight footprint for one active token, the weight-read lower bound alone would be about 14 GB per step. Real kernels, cache reuse, batch size, tensor parallelism, and KV traffic determine the observed bandwidth cost.

When profiling confirms this bandwidth ceiling, serving work should focus on bytes moved, cache residency, batch policy, and queueing behavior rather than only raw floating-point throughput.

The KV cache: saving state so you don't restart

Without caching, every decode step would recompute the Key and Value projections for earlier tokens before attending over the growing prefix. The KV cache stores those past Key and Value matrices, so we only need to compute them for the new token.

The KV cache acts as a handoff log between decode steps. Without it, the model would have to reread the entire prompt from the beginning every time it wanted to say the next word. With the KV cache, it remembers what it already processed and only computes fresh K/V state for the newest token.

Once KV state persists across decode steps, you also need to pack it efficiently instead of reserving one giant contiguous region per request. Reserved slabs waste HBM; paged blocks reuse slots as requests finish. Each decode step then reuses prefix KV state and adds one fresh pair for the newest token.

Memory cost of KV cache

The KV cache can become one of the largest consumers of GPU memory during inference, sometimes exceeding the model weights themselves for long contexts. That memory footprint drives capacity planning and the maximum batch size a given GPU can support.

A hand calculation makes the memory shape visible before the code. Suppose we're serving a long-context code assistant with a model that has 80 layers, uses Grouped Query Attention with 8 KV heads, and each head has dimension 128. For one request with a sequence length of 8,192 tokens, stored in FP16 (2 bytes per element):

• We need both K and V: that's a factor of 2
• One request, 8,192 tokens, 80 layers, 8 heads, head size 128, 2 bytes each
• Total bytes = 2 * 1 * 8,192 * 80 * 8 * 128 * 2 = 2,684,354,560 bytes
• Divide by 1024^3: that's about 2.5 GiB per request

Now scale that up. For a production batch of 64 concurrent requests at an 8K context window, that's about 160 GiB of KV cache alone. That scale makes techniques like Grouped Query Attention (GQA), which reduces the number of KV heads from num_heads to num_kv_heads, standard in modern models.^{[3]Reference 3GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.https://arxiv.org/abs/2305.13245}

This Python function generalizes that exact calculation. It takes the model's architectural parameters and returns the KV cache memory in GiB.

Throughput vs. latency trade-off

There's an inherent tension between maximizing system throughput and minimizing per-request latency.

Production tip: Monitor GPU KV-cache usage, prefill backlog, and decode queue depth together. High KV usage plus rising TTFT usually means memory pressure is capping concurrency. Low KV usage with idle compute means you're leaving throughput on the table.

The throughput, latency, cost triangle

Throughput and latency are two corners of a third constraint that the business cares about: cost per token. These three pull against each other, and picking where to sit on that triangle is the central job of an inference engineer.

Cost per token is simpler than it looks. If you rent a GPU at a fixed hourly rate and it sustains some number of tokens per second, then:

$\text{cost per token} = \frac{\text{GPU \$ per hour}}{\text{sustained tokens per second} \times 3600}$

Sustained throughput, not the sticker hourly rate, dominates the answer. A faster, pricier GPU can still be cheaper per token if its throughput rises faster than its price. A hand calculation shows the tradeoff. Suppose one GPU costs $3.00/hour and a well-batched deployment sustains 2,500 decode tokens/second across all active requests:

• Tokens per hour = 2,500 * 3,600 = 9,000,000
• Cost per token = $3.00 / 9,000,000 = $0.00000033
• Cost per million tokens = about $0.33

Now starve the batch. If under-configured batching or idle capacity drops sustained throughput to 250 tokens/second, the same GPU-hour spreads over one-tenth the tokens, so cost per million jumps to about $3.33. Utilization is a direct 10x multiplier on cost. Batching is more than a latency knob; it moves the cost corner of the triangle.

The triangle has a simple rule: you can usually optimize two corners hard, but the third drifts. Push batch size for throughput and cost, and tail latency rises. Cap batch size for tight latency SLOs, and your cost per token climbs because the GPU is underused. There is no single best operating point, only the one that fits your product's latency SLO at acceptable cost.

Production tip: Pick the operating point from the product SLO, then size hardware to it. An interactive assistant with a 500 ms TTFT budget can't run the same batch size as an overnight document-summarization job, even on identical GPUs. The summarization job can push batch size until cost per token bottoms out because no human is waiting on each token.

Chunked prefills

Long prefills (e.g., Retrieval-Augmented Generation (RAG), where retrieved documents are appended to the user's prompt, creating contexts of 10,000 tokens) can delay decode turns on a shared worker. The duration depends on model, kernel, hardware, and prompt length, but enough long prompts can noticeably worsen active streams in a multi-tenant service.

To reduce that interference, engineers can break large prefills into smaller chunks. Current vLLM V1 scheduling prioritizes pending decode requests, then uses remaining max_num_batched_tokens capacity for prefill work. If a prefill doesn't fit, the scheduler chunks it. This lets bounded prefill work share a batch with active decodes instead of forcing one huge uninterrupted prefill step. It doesn't guarantee a latency SLO when queues or kernels are already overloaded. Smaller token budgets can improve ITL, while larger budgets can improve TTFT and throughput.^{[4]Reference 4Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

Disaggregated inference

Disaggregated inference separates prefill and decode across worker pools rather than running both phases on one worker. Systems such as Splitwise and DistServe show when this can improve goodput: avoided interference must repay KV-transfer and coordination overhead.^{[5]Reference 5Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[2]Reference 2DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

The problem: conflicting optimization targets

Prefill and decode phases have opposite hardware needs:

• Prefill is often compute-heavy: Long prompts create large matrix operations over many positions
• Decode is often bandwidth-heavy: Small-batch token generation repeatedly reads weights and KV state

When both phases run on the same GPU, they interfere. A long prefill can "block" decode requests, causing head-of-line blocking where a massive prompt stalls generation for all other users.

Prefill-decode disaggregation

Serving systems often separate the phases:

1. Prefill cluster (compute-optimized): Dedicated prefill workers process incoming prompts in parallel. These nodes excel at the compute-heavy attention operations.

2. Decode cluster (bandwidth-optimized): Separate workers handle token generation. These nodes are tuned for the memory-bound sequential decoding loop and steady high-concurrency decode traffic.

3. KV cache handoff: After prefill completes, the KV cache is transferred over a fast interconnect from the prefill worker to a decode worker, which continues generation.

Benefits of disaggregation

• Can reduce head-of-line blocking: Long prefills no longer directly occupy decode workers
• Potentially right-sized hardware: Each phase can run on workers chosen for its measured bottleneck
• Separate scaling knobs: Prefill and decode clusters can scale independently based on workload patterns
• Potential efficiency gain: Separate pools can better match the two workload profiles when transfer overhead is acceptable

Disaggregation is a design pattern, not a mandatory default. The KV-transfer cost has to be lower than the queueing and interference it removes, which is why it becomes more attractive as prompts get longer and decode traffic gets denser.^{[5]Reference 5Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[2]Reference 2DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

Disaggregation also changes how you autoscale. Because prefill load tracks incoming prompt tokens and decode load tracks active generation, the two pools scale on different signals. A burst of long RAG prompts points toward more prefill capacity; a surge in concurrent streaming conversations points toward more decode capacity. Useful signals include queue depth, KV-cache utilization, and TTFT/TPOT percentiles, with cold-start time included because loading model weights onto a new worker isn't instant.

Batching strategies: scheduler slots

Naive batching strategies waste capacity because requests generate different numbers of tokens. Picture a GPU scheduler handling code-assistant requests: one needs a two-line answer, another streams a long patch explanation, and a third is still prefilled from a large file context.

Static batching: waiting for the whole batch

In static batching, the scheduler groups requests and keeps membership fixed for that run. If one request needs hundreds more decode steps, the shorter requests finish early but their slots often turn into padding or sit idle until the longest request finishes. In serving terms, requests are grouped into a batch and padded to the length of the longest active sequence. The timeline below shows shorter requests wasting compute cycles while the batch waits on the longest request.

The problem with static batching

Static batching creates two inefficiencies. First, the GPU is forced to process "padding tokens" that don't contribute to the final output, wasting compute cycles and memory bandwidth. Second, once shorter requests finish, their batch slots usually can't be reused until the scheduler rebuilds the batch around the longest surviving sequence. This degrades both latency and overall throughput, especially when request lengths vary widely.

Continuous batching: filling open decode slots

Continuous batching (introduced by Orca) operates at the iteration level.^{[6]Reference 6Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu} The scheduler releases one finished request, can pull the next queued request into the open slot, and keeps useful work flowing when demand exists.

In serving terms, instead of waiting for a whole batch cycle to finish, the scheduler can eject completed requests and insert new ones after every token generation step. See our continuous batching deep-dive for scheduling algorithms and preemption strategies. The timeline above shows slot reuse; its actual benefit depends on queued work, KV capacity, and latency policy.

Benefits of continuous batching

Continuous batching provides three useful advantages under mixed, queued workloads:

• Less slot waste: Finished requests can leave at iteration boundaries rather than remaining as padding or idle slots.
• Lower completion delay for short requests: A completed request need not wait for the longest request in a fixed batch, though queueing and large active batches can still worsen latency.
• Policy control: The scheduler can decide how to admit prefills alongside ongoing decode while respecting KV memory and latency SLOs.

Continuous batching uses a scheduling loop that manages active requests dynamically. This intentionally simplified sketch shows the shape of a continuous batcher. It takes a queue of incoming requests, admits work up to the maximum batch size, and then runs one decoding step for all active requests. A production scheduler also applies a token budget, chunks long prefills, and may co-batch prefill with decode.

Memory management: PagedAttention (vLLM)

Traditional KV-cache allocation is like reserving a full pallet position for every request, even if it only needs a small bin. PagedAttention is like a shared bin system that assigns fixed-size slots on demand: Request A gets slots 7, 2, and 5 (non-contiguous, but tracked by a block table). When a request finishes, its slots become available again. Paging sharply reduces worst-case reservation waste, but a partially filled final block and bookkeeping still consume memory.

The problem

KV cache is allocated per-request, but request lengths vary. Pre-allocating the maximum possible sequence length for every request wastes a massive amount of memory. For example, if the system allocates 4096 tokens per request by default:

PagedAttention solution

PagedAttention applies the operating system concept of virtual memory to KV cache management.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} Instead of contiguous physical memory, we divide the KV cache into fixed-size "blocks" (pages). The figure maps logical blocks to non-contiguous physical GPU memory through a block table.

Impact of PagedAttention

By avoiding large contiguous reservations and allocating fixed-size blocks on demand, PagedAttention lets the runtime fit more useful KV state into the same HBM budget.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} It doesn't eliminate all slack: each live request can still leave a partially filled last block, and the block table has overhead. In practice, it substantially reduces memory lost to worst-case preallocation.

Copy-on-write for shared blocks

PagedAttention's copy-on-write mechanism matters whenever multiple active continuations share the same prompt prefix. In the original vLLM setting, this is especially important for beam search and parallel sampling, where several continuations reuse the same prompt blocks before they diverge.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} This visual shows two continuations initially pointing to the same shared prefix blocks before branching.

Initially, the shared prefix blocks have a reference count greater than one. Appending new tokens usually allocates fresh blocks for each continuation. If a continuation needs to write into a block that's still shared, the runtime first clones that block so the other continuations keep their original view. That's what copy-on-write means here: share immutable prefix state aggressively, then split only when sequences diverge.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

Context parallelism and long-context serving

As context windows grow into the hundreds of thousands or millions of tokens, a single GPU often can't hold the full KV cache or attention working set for one request. Context Parallelism (CP) addresses this by splitting the input sequence itself across multiple GPUs.^{[8]Reference 8Ring Attention with Blockwise Transformers for Near-Infinite Context.https://arxiv.org/abs/2310.01889}

How context parallelism works

Instead of splitting layers (tensor parallelism) or batches (data parallelism), CP splits the sequence dimension:

1. A 1M token sequence is divided into N chunks (e.g., 250K tokens per GPU on 4 GPUs)
2. Each GPU processes its chunk independently during the prefill phase
3. Attention is computed using ring-style communication patterns to handle cross-chunk dependencies
4. The KV cache is distributed across the GPU cluster

This approach becomes useful once a single request's context no longer fits comfortably on one accelerator.

Ring attention for context parallelism

Modern implementations use ring attention (Liu et al., 2024) or similar distributed attention algorithms to manage communication overhead. GPUs form a logical ring. While each device computes blockwise attention for its local query block, Key and Value blocks circulate to the next device and arrive from the previous one. This can extend supported context length roughly linearly with the number of devices, at least until communication becomes the next bottleneck.^{[8]Reference 8Ring Attention with Blockwise Transformers for Near-Infinite Context.https://arxiv.org/abs/2310.01889}

Speculative decoding: draft then verify

Speculative decoding uses a small draft model to propose the next few tokens. The large target model checks the proposed span in a verification pass. If enough draft tokens survive acceptance, this can replace several target decode passes; if not, draft and verification work can lose to ordinary decoding.

Speculative decoding lets a large target model supervise a fast draft model. The draft model guesses the next 5 tokens. The target checks the span together. If the first 3 are accepted, one target verification pass can emit those accepted tokens plus a correction or continuation token. The accounting must still include the draft model's work.

To visualize this, consider how speculative decoding coordinates the interaction between the two models.^{[9]Reference 9Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192} We use a fast, small draft model to propose a sequence of multiple tokens. The large target model then verifies these proposed tokens in parallel, accepting correct ones and correcting any mistakes.

Speculative decoding relies on the asymmetric cost of the two paths. Drafting may be cheap enough, while verification can score all k draft positions in one target-model pass. If acceptance is high, one target pass can replace several ordinary target decode passes. Total latency still includes draft generation, verification, sampling, rejected-token correction, and kernel overhead.

This function shows one exact speculative step for a single sequence. It first samples k draft tokens from the small model, then runs the large target model once on [prompt + draft_tokens], and finally performs the accept/reject test from Leviathan et al. with residual resampling on the first mismatch.^{[9]Reference 9Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

Follow-on work such as EAGLE uses the target model's hidden states to predict future tokens instead of relying on a separately trained small draft model.^{[10]Reference 10EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.https://arxiv.org/abs/2401.15077} The important takeaway isn't that one speculative variant always wins. It's that these methods trade extra compute for fewer expensive target-model decode passes, so the real payoff depends on acceptance rate, hardware, and implementation overhead.

Hardware-aware optimization: quantization and precision

Inference performance isn't just about algorithms. Modern GPUs and specialized accelerators provide hardware-level features that change the efficiency equation.

Low-precision inference and quantization

Many serving stacks still use BF16/FP16 as a baseline, but lower-precision modes are increasingly common because they cut model-weight bandwidth and, in some systems, shrink the KV footprint enough to raise concurrency.^{[11]Reference 11FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433[12]Reference 12GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformershttps://arxiv.org/abs/2210.17323[13]Reference 13Quantized KV Cachehttps://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/}

• FP8 (8-bit floating point): Useful when your hardware and kernels support it, because it lowers bandwidth pressure while preserving more dynamic range than integer-only formats.^{[11]Reference 11FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}
• INT8/INT4-style weight quantization: Common for weight-only inference, where the target is fewer bytes reread on every decode step without fully quantizing the rest of the runtime.^{[12]Reference 12GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformershttps://arxiv.org/abs/2210.17323}
• KV-cache compression or quantization: Targets the other major memory consumer during long-context serving, which matters once the KV cache rather than weights caps concurrency.^{[14]Reference 14SnapKV: Compressing KV Cache by Selecting Global Attention Patterns.https://arxiv.org/abs/2403.19925[13]Reference 13Quantized KV Cachehttps://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/}

Quantization approaches include:

• Weight-only quantization: Keep activations in higher precision (BF16/FP16) but compress model weights to 4-8 bits
• KV-cache compression/quantization: Reduce KV bytes, or compress less useful KV state, when long contexts would otherwise cap batch size and residency

Common mistake: Beginners often think 4-bit quantization is just about saving disk space. The real win is reducing bytes moved during decode. Whether that turns into a large latency win still depends on kernels, dequant overhead, and quality constraints.

Inference-first silicon beyond GPUs

GPUs still dominate general-purpose LLM serving, but some deployments use inference-first accelerators with larger on-chip memory, dataflow execution, or more deterministic scheduling. Those accelerators usually come with a narrower software stack: you may get excellent latency or cost for a specific serving pattern, but at the price of custom compilation, fewer kernels, and less ecosystem flexibility.

When scaling inference breaks down

These symptoms show up in production logs and profiling traces. Each one points to a different bottleneck and fix.

"Training optimizations don't speed up my inference"

• Symptom: You upgrade math kernels or buy more peak FLOPs, but decode barely speeds up.
• Cause: Training is compute-bound, while decode is often memory-bound. If the GPU is already waiting on weights and KV reads, more math capacity does little.
• Fix: Profile bandwidth first. If HBM is near ceiling, prioritize batching, quantization, GQA, or KV-cache management before chasing more tensor-core throughput.

"Short requests are as slow as long ones"

• Symptom: Small chats wait almost as long as large ones even when the queue looks short.
• Cause: Static batching pads to the longest request and can't reuse finished slots until the batch cycle ends.
• Fix: Switch to continuous batching so completed requests leave immediately and queued work fills open slots on the next decode step.

"Speculative decoding made latency worse"

• Symptom: Draft-model serving added overhead, but target-model passes did not fall enough to repay it.
• Cause: Acceptance rate is too low or implementation overhead is too high. Wrong draft tokens force extra verification and correction work.
• Fix: Measure accepted tokens per draft span before rollout. If most proposals are rejected, keep standard decoding or use a stronger draft path.

"HBM usage grows almost linearly even when prefixes are shared"

• Symptom: HBM usage spikes even though many sessions start from the same long system prompt or retrieved prefix.
• Cause: The runtime isn't reusing cached prefix KV state across independent requests, or its isolation policy doesn't allow that reuse.
• Fix: Enable an explicit prefix-caching feature with an appropriate tenant and privacy policy. Copy-on-write can preserve shared blocks after reuse is established; it's not by itself a cross-request cache.

"Throughput keeps rising but users complain about slowness"

• Symptom: Aggregate TPS looks better, but TTFT or TPOT percentiles get worse and chat feels sluggish.
• Cause: Larger batches improve throughput while making active requests compete harder for bandwidth.
• Fix: Use SLO-aware scheduling. Cap batch size or split latency-sensitive traffic from background work, and watch throughput together with p95 or p99 latency.

Try it yourself: the VRAM calculator

Here's a practical check you can do with pen and paper or a short Python script.

• Problem: You want to serve a 7B-class model on a single GPU with 80 GiB of HBM. The model has 28 layers, uses Grouped Query Attention with 4 KV heads, head dimension 128, and you plan to use FP16 (2 bytes per element). Your average code-assistant request has a 4,096-token context. What's the maximum batch size you can support before the KV cache alone fills the GPU, assuming you need to leave 20 GiB for weights and runtime overhead?

• Hint: Start by computing the KV cache for one request, then see how many fit in the remaining 60 GiB.

Worked solution

For one request at 4,096 tokens:

Available memory for KV cache: 80 GiB total - 20 GiB reserved = 60 GiB

Maximum batch size = 60 GiB / 0.21875 GiB per request ≈ 274 requests (rounding the per-request figure to 0.22 GiB gives about 273, so treat this as a ballpark, not a precise count)

In practice, you'd run at a lower batch size to leave headroom for activation buffers, temporary tensors, allocator slack, and bursty long-context requests. A production engineer might cap this far below the arithmetic ceiling and monitor actual HBM usage.

Mastery check

What strong answers show

• Foundational: Why decode-heavy LLM serving is often memory-bandwidth bound even when peak FLOPs look huge.
• Intermediate: How prefill and decode differ, and why TTFT and TPOT need separate dashboards.
• Advanced: How KV-cache size scales with batch size, context length, layers, KV heads, head dimension, and precision.
• Advanced: Why static batching wastes slots, while continuous batching uses iteration-level admission to keep decode work flowing.
• Advanced: How PagedAttention uses virtual-memory-style blocks to pack KV state, and why cross-request prefix reuse requires explicit caching policy in addition to copy-on-write.
• Advanced: When speculative decoding, disaggregated inference, context parallelism, and quantization help, and when their overheads can erase the win.
• Advanced: How throughput, latency, and cost per token trade off, and how to pick an operating point from a product SLO instead of chasing one metric.

Follow-up questions

How does KV-cache size scale with sequence length and batch size?

KV cache stores key and value tensors from previous tokens across all layers, so memory grows linearly with sequence length, batch size, number of layers, number of KV heads, head dimension, and bytes per element. A useful formula is 2 * batch * seq_len * layers * kv_heads * head_dim * bytes. GQA reduces the kv_heads term, and KV-cache quantization reduces bytes.

When does speculative decoding hurt performance?

Speculative decoding hurts when the draft path is inaccurate or expensive enough that extra draft work and rejection handling cost more than the target-model passes saved. Low acceptance, poor kernel fusion, or tiny latency-sensitive batches can erase the win.

How does tensor parallelism interact with batching?

Tensor parallelism splits layer math across multiple GPUs, so every decode step includes collective communication such as all-reduce or all-gather. Larger batches can amortize that communication overhead, but small interactive batches may become communication-bound before they become compute-bound. Batching and tensor parallelism need to be tuned together.

What are the tradeoffs between throughput and latency?

Serving systems balance aggregate throughput against per-request latency. Continuous batching can increase total tokens per second, but larger batches also make active requests compete for the same memory bandwidth. Production schedulers need latency SLOs, not raw tokens/sec alone.

How do you estimate cost per token for a self-hosted model?

Take the GPU hourly rate, divide by sustained tokens per second times 3,600, then multiply by one million for cost per million tokens. The dominant variables are measured sustained throughput (driven by batching, quantization, and model size) and utilization. A GPU sitting idle multiplies cost per token directly, so a 5x cheaper sticker price can still lose if it delivers less than one-fifth the throughput.

A long RAG prefill is stalling active chat streams. Do you try chunked prefill or full disaggregation first?

Usually try chunked prefill first if the main problem is one large prompt monopolizing a shared worker for too long. Move to prefill-decode disaggregation when that interference remains large enough that separate pools and KV handoff beat their transfer and coordination overhead.

Course handoff

Serving capacity depends on phase behavior, memory bandwidth, KV-cache residency, scheduler policy, and precision choices. Those tools let you read a slow serving trace and separate prefill queueing, decode bandwidth pressure, fragmented KV memory, and speculative-decoding overhead.

Practice drill

Turn the capacity model into a worksheet for one production route:

1. Compare TTFT, TPOT, goodput, and cost per million tokens across at least three batch-size or concurrency settings.
2. Recompute cost per million tokens once with high utilization and once with latency-capped low utilization.
3. Mark which intervention you would test first: chunked prefill, continuous batching, PagedAttention, speculative decoding, quantization, or disaggregation.
4. Define one rollback trigger tied to user latency and one tied to cost or GPU utilization.

The useful artifact isn't a single "best" throughput number. It's a measured operating envelope that tells another engineer when to tune, when to split pools, and when to back out.