Scaling LLM Inference

Continuous batching keeps decode slots useful. The capacity question for scaling large language model (LLM) inference is harder: when requests, model weights, and KV state all compete for HBM, what limits serving concurrency?

A code assistant that answers "why is this test failing?" still generates every response one token at a time. It's not because the model is "thinking." During decode, the serving stack keeps rereading billions of model weights from GPU memory and consulting a growing KV cache, and that memory movement takes time. Reading the prompt happens in parallel; streaming the answer repeatedly revisits the same large weights and cache state.

Tie together prefill, decode, batching, KV-cache memory, PagedAttention, disaggregation, speculative decoding, and quantization so serving bottlenecks become measurable instead of mysterious. The thread running through all of them is one decision: where to sit on the throughput, latency, and cost triangle for your workload. For the scheduling loop behind this capacity model, see continuous batching.

Why does a serving engineer separate LLM generation into prefill and decode instead of treating inference as one uniform operation?

Answer

Prefill and decode stress different resources. Prefill processes the known prompt in parallel and is usually compute-heavy. Decode produces one token at a time, rereads weights and KV state repeatedly, and is usually memory-bandwidth bound. Good serving decisions depend on which phase is causing queueing, TTFT, or token-stream delay.

The two phases of generation

LLM inference is distinct from training because it consists of two very different computational phases: Prefill and Decode. Understanding this distinction is the first step to optimization.

A code-assistant request makes the two phases visible. When a user sends "Why did auth.test.ts fail after this refactor?", the system first has to read the entire prompt and attached context. That's the prefill phase. Then it starts answering, generating one token at a time: "The", "mock", "still", "uses", "the", "old", "schema." That's the decode phase.

Prefill: reading the prompt in one go

In the prefill phase, the model processes the entire user prompt in parallel. This is similar to training: the GPU receives a matrix of shape [batch_size, prompt_len, hidden_dim] and computes attention for all tokens simultaneously.

Because all the input tokens are known upfront, the attention mechanism can compute the interactions between every token in the prompt at once. This parallel processing allows the GPU to use its massive matrix multiplication engines efficiently. A long prefill usually dominates Time To First Token (TTFT), though TTFT also includes queueing and scheduling delay before the first output token is emitted. This visual shows how all tokens in the prompt are processed simultaneously to generate the first output token.

Key characteristics

• Often compute-heavy: Prefill exposes large matrix multiplications. FlashAttention keeps attention exact while reducing HBM traffic relative to materializing the full attention matrix; it doesn't make all attention IO linear in sequence length.^{[1]Reference 1FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.https://arxiv.org/abs/2205.14135}
• Parallel-friendly: Processing many prompt positions together can drive much higher tensor-core utilization than one-token decode. Whether it saturates compute depends on sequence shape, kernel, and hardware.
• Latency: Time usually grows with prompt length, and long prompts often dominate TTFT.

A user sends a 20,000-token RAG prompt and then receives a 50-token answer. Which phase likely dominates TTFT, and why?

Answer

The long prefill likely dominates TTFT because the model must process all 20,000 prompt tokens before emitting the first output token. Decode may still be memory-bound, but it only runs for about 50 generated tokens in this example.

Decode: answering one word at a time

Once the first token is generated, the model switches to autoregressive generation. It generates one token at a time, feeding it back as input for the next step.

Unlike the prefill phase, decoding can't be parallelized across tokens inside one request because each new token depends on the previous ones. The system is locked into a sequential, step-by-step loop. Users feel this as inter-token latency (ITL), also called time between tokens (TBT): the delay between visible streamed tokens. Time Per Output Token (TPOT) is closely related but isn't identical. DistServe, for example, defines TPOT as the average time to generate each output token after the first.^{[2]Reference 2DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

A rate such as tokens per second (TPS) is another view of decode speed. Be careful about the denominator: one stream's TPS isn't the same metric as aggregate tokens per second across a whole server. TTFT affects initial responsiveness, while ITL/TBT and TPOT describe the pace after streaming starts. During autoregressive decoding, each generated token becomes input for the next step.

Key characteristics

• Often memory-bound: A decode step needs model weights and the KV state used by attention. At small or latency-sensitive batches, repeated reads commonly make HBM bandwidth the ceiling; batching can raise arithmetic intensity by sharing weight reads across active requests.
• Low arithmetic intensity at small batches: The arithmetic intensity (FLOPs/byte, i.e., Floating Point Operations per byte of data loaded) can be low because the runtime moves large tensors for only one new position per sequence.

Why does batching help decode even though every request still needs its own next token?

Answer

The model weights can be read once for a larger batch of active tokens, so that expensive memory movement is amortized across requests. Batching doesn't remove the sequential dependency inside each request, but it raises useful work per weight read.

Why decode is memory-bound

Decode-heavy LLM serving is often memory-bandwidth bound, not compute-bound. In a compute-bound operation, the system is bottlenecked by the mathematical calculations it must perform. Training and long-prefill workloads typically expose much larger matrix operations than interactive decode, so they can drive compute hardware more effectively.

During token generation, the bottleneck often shifts toward memory movement. Each new token needs model weights and attention state, but contributes only one new position per active request. At modest decode batches, this produces low arithmetic intensity and makes HBM traffic a central constraint.

A 7 billion parameter model stored in 16-bit precision has weights of about 14 GB in decimal units. If one uncached decode step had to read that full weight footprint for one active token, the weight-read lower bound alone would be about 14 GB per step. Real kernels, cache reuse, batch size, tensor parallelism, and KV traffic determine the observed bandwidth cost.

When profiling confirms this bandwidth ceiling, serving work should focus on bytes moved, cache residency, batch policy, and queueing behavior rather than only raw floating-point throughput.

You profile decode and see tensor cores idle while HBM bandwidth is near saturation. Should you first buy more FLOPs or reduce memory traffic?

Answer

Reduce memory traffic first. The profile says the bottleneck is bandwidth, so higher FLOPs will sit idle too. Better first moves are larger effective batches, weight quantization, GQA/MQA, KV-cache compression, or better cache packing.

The KV cache: saving state so you don't restart

Without caching, every decode step would recompute the Key and Value projections for earlier tokens before attending over the growing prefix. The KV cache stores those past Key and Value matrices, so the model only computes them for the new token.

The KV cache acts as a handoff log between decode steps. Without it, the model would have to reread the entire prompt from the beginning every time it wanted to say the next word. With the KV cache, it remembers what it already processed and only computes fresh K/V state for the newest token.

Once KV state persists across decode steps, you also need to pack it efficiently instead of reserving one giant contiguous region per request. Reserved slabs waste HBM; paged blocks reuse slots as requests finish. Each decode step then reuses prefix KV state and adds one fresh pair for the newest token.

Memory cost of KV cache

The KV cache can become one of the largest consumers of GPU memory during inference, sometimes exceeding the model weights themselves for long contexts. That memory footprint drives capacity planning and the maximum batch size a given GPU can support.

A hand calculation makes the memory shape visible before the code. Suppose you're serving a long-context code assistant with a model that has 80 layers, uses Grouped Query Attention with 8 KV heads, and each head has dimension 128. For one request with a sequence length of 8,192 tokens, stored in FP16 (2 bytes per element):

• Both K and V are stored: that's a factor of 2
• One request, 8,192 tokens, 80 layers, 8 heads, head size 128, 2 bytes each
• Total bytes = 2 * 1 * 8,192 * 80 * 8 * 128 * 2 = 2,684,354,560 bytes
• Divide by 1024^3: that's about 2.5 GiB per request

Now scale that up. For a production batch of 64 concurrent requests at an 8K context window, that's about 160 GiB of KV cache alone. That scale makes techniques like Grouped Query Attention (GQA), which reduces the number of KV heads from num_heads to num_kv_heads, standard in modern models.^{[3]Reference 3GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.https://arxiv.org/abs/2305.13245}

This Python function generalizes that exact calculation. It takes the model's architectural parameters and returns the KV cache memory in GiB.

In the KV-cache formula, which terms can the model architecture reduce without shortening user context?

Answer

Grouped Query Attention or Multi-Query Attention reduces num_kv_heads. KV-cache quantization reduces dtype_bytes. Layer count and head dimension are architectural choices, but for a chosen model they are fixed at serving time. Batch size and sequence length are workload choices.

Throughput vs. latency trade-off

There's an inherent tension between maximizing system throughput and minimizing per-request latency.

Production tip: Monitor GPU KV-cache usage, prefill backlog, and decode queue depth together. High KV usage plus rising TTFT usually means memory pressure is capping concurrency. Low KV usage with idle compute means you're leaving throughput on the table. For stream feel, watch p95 ITL/TBT (and mean TPOT): chunked-prefill stalls are ITL spikes that can leave average TPOT almost fine.

If total tokens/sec improves after you raise batch size but users complain that streams feel slower, what metric did you optimize at the expense of what metric?

Answer

You improved aggregate throughput at the expense of per-request latency, especially TTFT and p95 ITL/TBT (mean TPOT can still look acceptable). Serving dashboards need both system tokens/sec and user-facing latency percentiles.

The throughput, latency, cost triangle

Throughput and latency are two corners of a third constraint that the business cares about: cost per token. These three pull against each other, and picking where to sit on that triangle is the central job of an inference engineer.

Cost per token is simpler than it looks. If you rent a GPU at a fixed hourly rate and it sustains some number of tokens per second, then:

$\text{cost per token} = \frac{\text{GPU \$ per hour}}{\text{sustained tokens per second} \times 3600}$

Sustained throughput, not the sticker hourly rate, dominates the answer. A faster, pricier GPU can still be cheaper per token if its throughput rises faster than its price. A hand calculation shows the tradeoff. Suppose one GPU costs $3.00/hour and a well-batched deployment sustains 2,500 decode tokens/second across all active requests:

• Tokens per hour = 2,500 * 3,600 = 9,000,000
• Cost per token = $3.00 / 9,000,000 = $0.00000033
• Cost per million tokens = about $0.33

Now starve the batch. If under-configured batching or idle capacity drops sustained throughput to 250 tokens/second, the same GPU-hour spreads over one-tenth the tokens, so cost per million jumps to about $3.33. Utilization is a direct 10x multiplier on cost. Batching is more than a latency knob; it moves the cost corner of the triangle.

The triangle has a simple rule: you can usually optimize two corners hard, but the third drifts. Push batch size for throughput and cost, and tail latency rises. Cap batch size for tight latency SLOs, and your cost per token climbs because the GPU is underused. There is no single best operating point, only the one that fits your product's latency SLO at acceptable cost.

Production tip: Pick the operating point from the product SLO, then size hardware to it. An interactive assistant with a 500 ms TTFT budget can't run the same batch size as an overnight document-summarization job, even on identical GPUs. The summarization job can push batch size until cost per token bottoms out because no human is waiting on each token.

Two teams serve the same model on the same GPU but report very different cost per million tokens. The interactive team is 5x more expensive. Why is that not necessarily a misconfiguration?

Answer

Interactive serving caps batch size to protect TTFT and TPOT, so the GPU runs below its throughput ceiling and each GPU-hour spreads over fewer tokens. The batch team pushes batch size until throughput saturates, lowering cost per token. Both can be correct operating points for their SLOs.

Chunked prefills

Long prefills (e.g., Retrieval-Augmented Generation (RAG), where retrieved documents are appended to the user's prompt, creating contexts of 10,000 tokens) can delay decode turns on a shared worker. The duration depends on model, kernel, hardware, and prompt length, but enough long prompts can noticeably worsen active streams in a multi-tenant service.

To reduce that interference, engineers can break large prefills into smaller chunks. Current vLLM V1 scheduling prioritizes pending decode requests, then uses remaining max_num_batched_tokens capacity for prefill work. If a prefill doesn't fit, the scheduler chunks it. This lets bounded prefill work share a batch with active decodes instead of forcing one huge uninterrupted prefill step. It doesn't guarantee a latency SLO when queues or kernels are already overloaded. Smaller token budgets can improve ITL, while larger budgets can improve TTFT and throughput.^{[4]Reference 4Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

What does chunked prefill sacrifice, and what does it protect?

Answer

It can increase TTFT for the long prompt because that prompt is processed in pieces. It can improve inter-token latency for already-active decode requests by bounding the prefill work admitted alongside them.

Disaggregated inference

Disaggregated inference separates prefill and decode across worker pools rather than running both phases on one worker. Systems such as Splitwise and DistServe show when this can improve goodput: avoided interference must repay KV-transfer and coordination overhead.^{[5]Reference 5Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[2]Reference 2DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

The problem: conflicting optimization targets

Prefill and decode phases have opposite hardware needs:

• Prefill is often compute-heavy: Long prompts create large matrix operations over many positions
• Decode is often bandwidth-heavy: Small-batch token generation repeatedly reads weights and KV state

When both phases run on the same GPU, they interfere. A long prefill can "block" decode requests, causing head-of-line blocking where a massive prompt stalls generation for all other users.

Prefill-decode disaggregation

Serving systems often separate the phases:

1. Prefill cluster (compute-optimized): Dedicated prefill workers process incoming prompts in parallel. These nodes excel at the compute-heavy attention operations.

2. Decode cluster (bandwidth-optimized): Separate workers handle token generation. These nodes are tuned for the memory-bound sequential decoding loop and steady high-concurrency decode traffic.

3. KV cache handoff: After prefill completes, the KV cache is transferred over a fast interconnect from the prefill worker to a decode worker, which continues generation.

Benefits of disaggregation

• Can reduce head-of-line blocking: Long prefills no longer directly occupy decode workers
• Potentially right-sized hardware: Each phase can run on workers chosen for its measured bottleneck
• Separate scaling knobs: Prefill and decode clusters can scale independently based on workload patterns
• Potential efficiency gain: Separate pools can better match the two workload profiles when transfer overhead is acceptable

Disaggregation is a design pattern, not a mandatory default. The KV-transfer cost has to be lower than the queueing and interference it removes, which is why it becomes more attractive as prompts get longer and decode traffic gets denser.^{[5]Reference 5Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[2]Reference 2DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

Disaggregation also changes how you autoscale. Because prefill load tracks incoming prompt tokens and decode load tracks active generation, the two pools scale on different signals. A burst of long RAG prompts points toward more prefill capacity; a surge in concurrent streaming conversations points toward more decode capacity. Useful signals include queue depth, KV-cache utilization, and TTFT/TPOT percentiles, with cold-start time included because loading model weights onto a new worker isn't instant.

When can prefill-decode disaggregation make latency worse instead of better?

Answer

It can hurt when KV-cache transfer, network coordination, or extra scheduling overhead costs more than the head-of-line blocking it removes. Short prompts, light traffic, or slow interconnects often favor a simpler monolithic worker.

Batching strategies: scheduler slots

Naive batching strategies waste capacity because requests generate different numbers of tokens. Picture a GPU scheduler handling code-assistant requests: one needs a two-line answer, another streams a long patch explanation, and a third is still prefilled from a large file context.

Static batching: waiting for the whole batch

In static batching, the scheduler groups requests and keeps membership fixed for that run. If one request needs hundreds more decode steps, the shorter requests finish early but their slots often turn into padding or sit idle until the longest request finishes. In serving terms, requests are grouped into a batch and padded to the length of the longest active sequence. The timeline below shows shorter requests wasting compute cycles while the batch waits on the longest request.

The problem with static batching

Static batching creates two inefficiencies. First, the GPU is forced to process "padding tokens" that don't contribute to the final output, wasting compute cycles and memory bandwidth. Second, once shorter requests finish, their batch slots usually can't be reused until the scheduler rebuilds the batch around the longest surviving sequence. This degrades both latency and overall throughput, especially when request lengths vary widely.

Why is static batching especially painful when output lengths vary widely?

Answer

Short requests finish early but their slots can't be reused until the batch cycle ends, so the system either processes padding or leaves capacity idle while waiting for long requests.

Continuous batching: filling open decode slots

Continuous batching (introduced by Orca) operates at the iteration level.^{[6]Reference 6Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu} The scheduler releases one finished request, can pull the next queued request into the open slot, and keeps useful work flowing when demand exists.

In serving terms, instead of waiting for a whole batch cycle to finish, the scheduler can eject completed requests and insert new ones after every token generation step. See our continuous batching deep-dive for scheduling algorithms and preemption strategies. The timeline above shows slot reuse; its actual benefit depends on queued work, KV capacity, and latency policy.

Benefits of continuous batching

Continuous batching provides three useful advantages under mixed, queued workloads:

• Less slot waste: Finished requests can leave at iteration boundaries rather than remaining as padding or idle slots.
• Lower completion delay for short requests: A completed request need not wait for the longest request in a fixed batch, though queueing and large active batches can still worsen latency.
• Policy control: The scheduler can decide how to admit prefills alongside ongoing decode while respecting KV memory and latency SLOs.

What does "iteration-level scheduling" mean in continuous batching?

Answer

The scheduler can change batch membership after each decode iteration. Finished requests leave immediately, queued requests enter open slots, and active requests continue from their stored KV state.

Continuous batching uses a scheduling loop that manages active requests dynamically. This intentionally simplified sketch shows the shape of a continuous batcher. It takes a queue of incoming requests, admits work up to the maximum batch size, and then runs one decoding step for all active requests. A production scheduler also applies a token budget, chunks long prefills, and may co-batch prefill with decode.

Memory management: PagedAttention (vLLM)

Traditional KV-cache allocation reserves one contiguous maximum-length region per request, even when most requests finish early. PagedAttention instead assigns fixed-size physical blocks on demand: Request A can use blocks 7, 2, and 5 while a block table preserves logical token order. Finished requests return their blocks to the allocator. Paging sharply reduces worst-case reservation waste, but a partially filled final block and bookkeeping still consume memory.

What problem does PagedAttention solve that continuous batching alone doesn't solve?

Answer

Continuous batching decides which requests run each iteration. PagedAttention decides how their KV cache is physically packed in GPU memory. Without paging, active requests can still waste memory through large contiguous reservations and fragmentation.

The problem

KV cache is allocated per-request, but request lengths vary. Pre-allocating the maximum possible sequence length for every request wastes a massive amount of memory. For example, if the system allocates 4096 tokens per request by default:

PagedAttention solution

PagedAttention applies the operating system concept of virtual memory to KV cache management.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} Instead of contiguous physical memory, the system divides the KV cache into fixed-size "blocks" (pages). The figure maps logical blocks to non-contiguous physical GPU memory through a block table.

Impact of PagedAttention

By avoiding large contiguous reservations and allocating fixed-size blocks on demand, PagedAttention lets the runtime fit more useful KV state into the same HBM budget.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} It doesn't eliminate all slack: each live request can still leave a partially filled last block, and the block table has overhead. In practice, it substantially reduces memory lost to worst-case preallocation.

Copy-on-write for shared blocks

PagedAttention's copy-on-write mechanism matters whenever multiple active continuations share the same prompt prefix. In the original vLLM setting, this is especially important for beam search and parallel sampling, where several continuations reuse the same prompt blocks before they diverge.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} This visual shows two continuations initially pointing to the same shared prefix blocks before branching.

Initially, the shared prefix blocks have a reference count greater than one. Appending new tokens usually allocates fresh blocks for each continuation. If a continuation needs to write into a block that's still shared, the runtime first clones that block so the other continuations keep their original view. That's what copy-on-write means here: share immutable prefix state aggressively, then split only when sequences diverge.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

Why does copy-on-write matter for beam search, and what else is required to share prefixes across independent requests?

Answer

Beam-search or parallel-sampling continuations can share already-created KV blocks until their paths diverge. Sharing a system prompt across independent requests also needs an explicit prefix-cache lookup and an isolation policy; copy-on-write alone does not discover that reuse.

Context parallelism and long-context serving

As context windows grow into the hundreds of thousands or millions of tokens, a single GPU often can't hold the full KV cache or attention working set for one request. Context Parallelism (CP) addresses this by splitting the input sequence itself across multiple GPUs.^{[8]Reference 8Ring Attention with Blockwise Transformers for Near-Infinite Context.https://arxiv.org/abs/2310.01889}

How context parallelism works

Instead of splitting layers (tensor parallelism) or batches (data parallelism), CP splits the sequence dimension:

1. A 1M token sequence is divided into N chunks (e.g., 250K tokens per GPU on 4 GPUs)
2. Each GPU processes its chunk independently during the prefill phase
3. Attention is computed using ring-style communication patterns to handle cross-chunk dependencies
4. The KV cache is distributed across the GPU cluster

This approach becomes useful once a single request's context no longer fits comfortably on one accelerator.

Ring attention for context parallelism

Modern implementations use ring attention (Liu et al., 2024) or similar distributed attention algorithms to manage communication overhead. GPUs form a logical ring. While each device computes blockwise attention for its local query block, Key and Value blocks circulate to the next device and arrive from the previous one. This can extend supported context length roughly linearly with the number of devices, at least until communication becomes the next bottleneck.^{[8]Reference 8Ring Attention with Blockwise Transformers for Near-Infinite Context.https://arxiv.org/abs/2310.01889}

What bottleneck does context parallelism target, and what bottleneck can it introduce?

Answer

It targets per-request context and KV memory that no longer fits on one GPU. It can introduce communication overhead because devices must exchange partial attention state across sequence shards.

Speculative decoding: draft then verify

Speculative decoding uses a small draft model to propose the next few tokens. The large target model checks the proposed span in a verification pass. If enough draft tokens survive acceptance, this can replace several target decode passes; if not, draft and verification work can lose to ordinary decoding.

Speculative decoding lets a large target model supervise a fast draft model. The draft model guesses the next 5 tokens. The target checks the span together. If the first 3 are accepted, one target verification pass can emit those accepted tokens plus a correction or continuation token. The accounting must still include the draft model's work.

To visualize this, consider how speculative decoding coordinates the interaction between the two models.^{[9]Reference 9Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192} A fast, small draft model proposes a sequence of multiple tokens. The large target model then verifies these proposed tokens in parallel, accepting correct ones and correcting any mistakes.

Speculative decoding relies on the asymmetric cost of the two paths. Drafting may be cheap enough, while verification can score all k draft positions in one target-model pass. If acceptance is high, one target pass can replace several ordinary target decode passes. Total latency still includes draft generation, verification, sampling, rejected-token correction, and kernel overhead.

This function shows one exact speculative step for a single sequence. It first samples k draft tokens from the small model, then runs the large target model once on [prompt + draft_tokens], and finally performs the accept/reject test from Leviathan et al. with residual resampling on the first mismatch.^{[9]Reference 9Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

Follow-on work such as EAGLE uses the target model's hidden states to predict future tokens instead of relying on a separately trained small draft model.^{[10]Reference 10EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.https://arxiv.org/abs/2401.15077} The important takeaway isn't that one speculative variant always wins. It's that these methods trade extra compute for fewer expensive target-model decode passes, so the real payoff depends on acceptance rate, hardware, and implementation overhead.

Why can speculative decoding be exact yet still slower than normal decoding?

Answer

The accept/reject rule can preserve the target model distribution, but performance depends on acceptance rate and overhead. If the draft model is wrong too often or the implementation adds too much work, the saved target-model passes don't repay the extra draft and verification cost.

Hardware-aware optimization: quantization and precision

Inference performance isn't just about algorithms. Modern GPUs and specialized accelerators provide hardware-level features that change the efficiency equation.

Low-precision inference and quantization

Many serving stacks still use BF16/FP16 as a baseline, but lower-precision modes are increasingly common because they cut model-weight bandwidth and, in some systems, shrink the KV footprint enough to raise concurrency.^{[11]Reference 11FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433[12]Reference 12GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformershttps://arxiv.org/abs/2210.17323[13]Reference 13Quantized KV Cachehttps://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/}

• FP8 (8-bit floating point): Useful when your hardware and kernels support it, because it lowers bandwidth pressure while preserving more dynamic range than integer-only formats.^{[11]Reference 11FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}
• INT8/INT4-style weight quantization: Common for weight-only inference, where the target is fewer bytes reread on every decode step without fully quantizing the rest of the runtime.^{[12]Reference 12GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformershttps://arxiv.org/abs/2210.17323}
• KV-cache compression or quantization: Targets the other major memory consumer during long-context serving, which matters once the KV cache rather than weights caps concurrency.^{[14]Reference 14SnapKV: Compressing KV Cache by Selecting Global Attention Patterns.https://arxiv.org/abs/2403.19925[13]Reference 13Quantized KV Cachehttps://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/}

Quantization approaches include:

• Weight-only quantization: Keep activations in higher precision (BF16/FP16) but compress model weights to 4-8 bits
• KV-cache compression/quantization: Reduce KV bytes, or compress less useful KV state, when long contexts would otherwise cap batch size and residency

Common mistake: Beginners often think 4-bit quantization is just about saving disk space. The real win is reducing bytes moved during decode. Whether that turns into a large latency win still depends on kernels, dequant overhead, and quality constraints.

Why does weight-only INT4 often help decode latency more directly than prefill latency?

Answer

Decode repeatedly streams model weights for small per-token work, so fewer bytes per weight directly reduce bandwidth pressure. Prefill uses larger matrix operations and can be more compute-heavy, so weight compression is not always the dominant lever there.

Inference-first silicon beyond GPUs

GPUs still dominate general-purpose LLM serving, but some deployments use inference-first accelerators with larger on-chip memory, dataflow execution, or more deterministic scheduling. Those accelerators usually come with a narrower software stack: you may get excellent latency or cost for a specific serving pattern, but at the price of custom compilation, fewer kernels, and less ecosystem flexibility.

When scaling inference breaks down

These symptoms show up in production logs and profiling traces. Each one points to a different bottleneck and fix.

"Training optimizations don't speed up my inference"

• Symptom: You upgrade math kernels or buy more peak FLOPs, but decode barely speeds up.
• Cause: Training is compute-bound, while decode is often memory-bound. If the GPU is already waiting on weights and KV reads, more math capacity does little.
• Fix: Profile bandwidth first. If HBM is near ceiling, prioritize batching, quantization, GQA, or KV-cache management before chasing more tensor-core throughput.

"Short requests are as slow as long ones"

• Symptom: Small chats wait almost as long as large ones even when the queue looks short.
• Cause: Static batching pads to the longest request and can't reuse finished slots until the batch cycle ends.
• Fix: Switch to continuous batching so completed requests leave immediately and queued work fills open slots on the next decode step.

"Speculative decoding made latency worse"

• Symptom: Draft-model serving added overhead, but target-model passes did not fall enough to repay it.
• Cause: Acceptance rate is too low or implementation overhead is too high. Wrong draft tokens force extra verification and correction work.
• Fix: Measure accepted tokens per draft span before rollout. If most proposals are rejected, keep standard decoding or use a stronger draft path.

"HBM usage grows almost linearly even when prefixes are shared"

• Symptom: HBM usage spikes even though many sessions start from the same long system prompt or retrieved prefix.
• Cause: The runtime isn't reusing cached prefix KV state across independent requests, or its isolation policy doesn't allow that reuse.
• Fix: Enable an explicit prefix-caching feature with an appropriate tenant and privacy policy. Copy-on-write can preserve shared blocks after reuse is established; it's not by itself a cross-request cache.

"Throughput keeps rising but users complain about slowness"

• Symptom: Aggregate TPS looks better, but TTFT or p95 ITL/TBT (and mean TPOT) get worse and chat feels sluggish.
• Cause: Larger batches improve throughput while making active requests compete harder for bandwidth.
• Fix: Use SLO-aware scheduling. Cap batch size or split latency-sensitive traffic from background work, and watch throughput together with p95 ITL/TBT or p99 latency, not mean TPOT alone.

A dashboard shows high TTFT, rising prefill backlog, stable TPOT, and moderate KV memory. Which subsystem should you inspect first?

Answer

Inspect prefill scheduling and admission first. Stable TPOT suggests decode is not currently degrading, while high TTFT plus prefill backlog points to prompt processing or queueing before the first token.

Try it yourself: the VRAM calculator

Here's a practical check you can do with pen and paper or a short Python script.

• Problem: You want to serve a 7B-class model on a single GPU with 80 GiB of HBM. The model has 28 layers, uses Grouped Query Attention with 4 KV heads, head dimension 128, and you plan to use FP16 (2 bytes per element). Your average code-assistant request has a 4,096-token context. What's the maximum batch size you can support before the KV cache alone fills the GPU, assuming you need to leave 20 GiB for weights and runtime overhead?

• Hint: Start by computing the KV cache for one request, then see how many fit in the remaining 60 GiB.

Worked solution

For one request at 4,096 tokens:

Available memory for KV cache: 80 GiB total - 20 GiB reserved = 60 GiB

Maximum batch size = 60 GiB / 0.21875 GiB per request ≈ 274 requests (rounding the per-request figure to 0.22 GiB gives about 273, so treat this as a ballpark, not a precise count)

In practice, you'd run at a lower batch size to leave headroom for activation buffers, temporary tensors, allocator slack, and bursty long-context requests. A production engineer might cap this far below the arithmetic ceiling and monitor actual HBM usage.

Why is the arithmetic maximum batch size from a KV-cache formula not a safe production setting?

Answer

It ignores activation buffers, temporary tensors, allocator fragmentation, model weights, runtime overhead, prefix sharing behavior, and traffic bursts with longer contexts. Production capacity should use measured headroom and latency SLOs, not a closed-form memory ceiling alone.