Inference: TTFT, TPS & KV Cache

The previous chapter showed how multi-agent systems turn one product action into planning, retrieval, tool calls, review, and several model calls. This chapter zooms into one of those calls. Once a request reaches an inference engine, three terms matter immediately: time to first token (TTFT), tokens per second (TPS), and the key-value (KV) cache that grows with the conversation.

When you send a message to ChatGPT, you notice something peculiar about how the response appears. There's often a brief pause, then the first visible text appears, followed by the rest streaming out in small chunks. Why that initial pause? Why does it stream token by token instead of appearing instantly? And why do longer conversations eventually feel slower?

This behavior isn't a quirk of the interface. It's the physics of LLM inference: the process of running a trained model to generate text. Understanding these mechanics (two-phase nature of generation, memory bottlenecks, and how engineers measure performance) is important for anyone building or optimizing production AI systems. Longer conversations often feel slower for a concrete reason: each decode step has to read a growing KV cache on top of the model weights.

Key insight: LLM inference has two distinct phases (prefill and decode) that usually emphasize different hardware bottlenecks. Knowing which phase is the bottleneck, and why, is the foundation for major optimizations in this space, from Key-Value (KV) cache management to continuous batching.

---

The two phases of LLM inference

Every LLM request goes through two distinct computational phases with fundamentally different hardware bottlenecks:

Analogy (fulfillment desk): Prefill is like reading the entire order history, return policy, and carrier notes at once before the first reply. It's intense upfront work but highly parallelizable. Decode is like scanning one outbound box at a time in strict order: each new token waits for the previous token, and the bottleneck is how fast the system can fetch stored state from memory.

The diagram below illustrates the sequential dependency between the highly parallel prefill phase and the autoregressive decode phase:

Phase 1: Prefill (processing the prompt)

In an unchunked baseline, the model processes the input prompt in parallel during prefill. Prompt tokens participate in large matrix operations, so the GPU can run dense matrix multiplies efficiently. For long prompts on modern accelerators, this phase is usually compute-bound, limited more by available FLOPs than by memory bandwidth. Later in this lesson, chunked prefill deliberately divides that work for scheduling reasons. A toy trace looks like this:

The time from request to the first output token is called TTFT (Time to First Token). For a user staring at a chat interface, this is the most noticeable latency. Do not copy one generic latency target into every product; establish a service-level objective (SLO) from the interaction mode and measured user tolerance.

Phase 2: Decode (generating output tokens)

After producing the first token, the model generates subsequent tokens one at a time, autoregressively. Each new token requires a forward pass through the entire model, but only processes that single new token (reusing cached K/V from all previous tokens). The new token is appended to the running sequence, added to the KV cache, and fed back into the model to predict the next token. The trace below shows how the response grows:

For a single decode stream, this phase is usually memory-bandwidth bound, not compute-bound. The bottleneck is reading the model weights and KV cache from GPU HBM (High Bandwidth Memory) for each token. The matrix multiplications are thin relative to the amount of data that must be moved, so the GPU's arithmetic units often spend more time waiting for bytes than doing math. As the response grows, the attention kernel also has to read a larger cached prefix, so per-token latency tends to rise with sequence length even when the model weights stay fixed.

The following small timeline separates first-token latency from decode cadence. It is intentionally a measurement exercise, not a model benchmark.

The arithmetic intensity explanation

The key difference between the two phases is arithmetic intensity: the number of floating-point operations (FLOPs) the GPU can perform per byte of data it must fetch from high-bandwidth memory (HBM).

Why prefill has high arithmetic intensity. During prefill the model loads each weight matrix (the ~140 GB of parameters) once and reuses it across the entire prompt batch of $N$ tokens. The expensive memory traffic is amortized over a large number of matrix multiplications. The GPU's tensor cores stay saturated with useful work; the bottleneck becomes raw compute throughput (TFLOPS, trillions of floating-point operations per second).

Why decode has low arithmetic intensity. For each new output token, the model effectively streams through the weight tensor (~140 GB for a 70B-class BF16/FP16 model) to perform what is close to a matrix-vector product when the effective batch is small. The number of FLOPs per byte loaded collapses. The GPU's arithmetic units spend much of their time waiting for the next wave of weights and KV cache entries to arrive from HBM. Memory bandwidth (TB/s) therefore becomes the limiting factor.

The roofline model^[1] makes this concrete: a kernel's achievable throughput is capped by either peak compute or by (memory bandwidth × arithmetic intensity), whichever is lower. Below a hardware-specific intensity threshold (the "ridge point"), you are bandwidth-bound; above it, compute-bound. Prefill often sits above that threshold, while low-batch decode often sits below it. An H100 SXM GPU has 80 GB of HBM and peak HBM bandwidth of 3.35 TB/s.^[2] A 70B-class model in BF16/FP16 (16-bit floating-point formats) needs roughly 138-140 GB for weights alone, so it cannot be served unsharded on one H100-80GB.^[3] That asymmetry is why input/output-aware (IO-aware) kernels like FlashAttention^[4] matter for long prefills, while PagedAttention^[5] focuses on fitting and reusing the KV cache efficiently during serving.

---

Key performance metrics

To evaluate and optimize an inference system, engineers rely on four standard metrics that capture different parts of the user experience and system throughput. Balancing these metrics often involves direct tradeoffs.

TTFT (time to first token)

TTFT measures how long it takes before the first output token appears.^[6] At the model-kernel level, TTFT is dominated by the prefill phase. In a real serving stack, end-to-end TTFT also includes tokenization, queueing, scheduling, and network overhead. Once those are under control, TTFT usually scales roughly linearly with prompt length for long prompts.

This metric is highly visible to users. It's critical for interactive applications like chat interfaces, voice assistants, and real-time code completion, where a delay of even a second can feel sluggish.

TPS (tokens per second)

TPS (also called "decode throughput") measures how fast the model generates subsequent output tokens after the first token is produced. For low-batch decode, memory bandwidth is often the dominant constraint. Exact results depend on model architecture, quantization, parallelism, engine, context length, and batch shape; measure the deployed configuration rather than using a generic tokens-per-second claim. Production systems use batched inference, often with continuous batching,^[7] to improve aggregate throughput across requests.

ITL (inter-token latency)

ITL represents the time elapsed between generating consecutive output tokens.^[6] For a single request running in isolation, ITL is simply the inverse of TPS ($ITL = 1/TPS$).

In interactive UIs, ITL above roughly 100 milliseconds often starts to feel choppy. Under heavy batching, ITL can increase because more requests are contending for the same GPU resources.

TPOT (time per output token)

While ITL is the individual gap between adjacent streamed tokens, TPOT is the average time per generated output token after the first token. In practice, TPOT is often computed as the mean of a request's ITL values or as an aggregate benchmark statistic across requests.^[6]

Monitoring both metrics matters: ITL exposes jitter and stalls in the stream, while TPOT summarizes overall decode pacing. Under load, prefill interruptions, scheduling, and batching contention can make both worse.

The four metrics are summarized below:

An alert can route to the relevant investigation path without pretending to diagnose root cause by itself:

Back-of-the-envelope: a bandwidth upper bound

For a single decode stream, TPS is governed by memory bandwidth. To generate one token, the GPU roughly has to stream the model weights from HBM. A rough estimate:

$\text{Max TPS} \approx \frac{\text{HBM Bandwidth}}{\text{Model size (bytes)}}$

A 70B BF16/FP16 weight tensor does not fit on one H100-80GB. Suppose it is tensor-parallel sharded across four H100 SXM GPUs. Each GPU holds about one quarter of the 140 GB weight tensor and can read its shard at up to 3.35 TB/s; equivalently, the four devices offer an idealized aggregate 13.4 TB/s before communication and runtime losses.^[2][3]

$\text{Ideal shard-read bound} \approx \frac{4 \times 3{,}350 \text{ GB/s}}{140 \text{ GB}} \approx 96 \text{ tokens/sec}$

This is an optimistic bandwidth bound, not a throughput promise. It omits tensor-parallel communication, KV cache reads during attention, activation memory, kernel overhead, less-than-peak sustained bandwidth, and scheduling. Benchmark the selected engine and parallelism layout to get real TPS. Batched inference can improve aggregate throughput because multiple sequences share weight reads across concurrent decode work.

Research note: This memory-bandwidth bound is fundamental to transformer inference and is why many serving optimizations (PagedAttention^[5], continuous batching^[7], quantization) attack the same bottleneck: reducing bytes moved per token or increasing effective HBM bandwidth.

---

The KV cache: the dynamic capacity bottleneck

After weights and runtime buffers are resident, the limit on how many active sequences a deployment can admit is often remaining memory capacity, specifically the memory required for their KV cache.

What is the KV cache?

During attention, each layer computes Key and Value projections for every token. Without caching, each decode step would have to rerun the full prefix through the model and recompute old K/V tensors again and again. That repeated work gets expensive fast as the sequence grows.

Analogy (shipment trace): The KV cache is like a shipment trace you build while processing an order conversation. For each token already processed, the model stores key routing facts (K) and useful details (V). When the next token references earlier context, the model reads that trace instead of re-deriving the whole prompt from scratch. The trace grows with each token, and its size determines how many concurrent conversations fit in GPU memory.

The KV cache stores these K and V tensors. As the sequence grows, the KV cache accumulates data for each token to avoid redundant calculations. This step-by-step accumulation allows the model to compute attention for only the newest token against the historical cache. The trace below shows how the cache expands with each generated token:

To manage this growing memory dynamically without fragmentation, systems like vLLM use PagedAttention^[5], which divides the KV cache into fixed-size blocks (pages) similar to operating system virtual memory.

KV cache memory formula

For a single sequence:

$\text{KV Cache} = 2 \times L \times n_{kv} \times d_h \times s \times b$

Reading the formula: for every layer ($L$), every KV head ($n_{kv}$), every position in the sequence ($s$), we store a Key vector and a Value vector (the "2") of dimension $d_h$, each taking $b$ bytes. Multiply it all together and this cache can easily reach gigabytes for long sequences.

Where:

• $L$ = number of layers
• $n_{kv}$ = number of KV heads (reduced with GQA/MQA (Grouped-Query/Multi-Query Attention)^[8])
• $d_h$ = head dimension
• $s$ = sequence length
• $b$ = bytes per element (2 for FP16, 1 for an 8-bit cache such as INT8 or FP8)

Concrete example: 70B-class GQA model

$$\begin{aligned} \text{KV Cache} &= 2 \times 80 \times 8 \times 128 \times 4096 \times 2 \\ &\approx \textbf{1.34 GB} \; (\approx \textbf{1.25 GiB}) \textbf{ per sequence} \end{aligned}$$

Here's the breakdown: the formula multiplies $2$ (for K and V) by $80$ layers, $8$ KV heads, a $128$ head dimension, a $4096$ sequence length, and $2$ bytes per value (for FP16). This matches the common 64-query-head / 8-KV-head GQA geometry used by many 70B-class models. With GQA, this is 8× smaller than it would be with standard Multi-Head Attention (MHA). Without GQA, the same geometry would need about 10.74 GB (10.0 GiB) per sequence.

Production note: Serving a 70B model to 100 concurrent users requires precise KV cache budgeting: model weights (~138-140 GB BF16/FP16) + KV cache (1.34 GB × 100 users = 134 GB) = about 274 GB of raw footprint before allocator overhead, runtime buffers, and communication memory. On paper that fits across 4× H100-80GB with tensor parallelism, but it doesn't leave unlimited headroom.

Try it yourself: A colleague says you can serve a 7B model (32 layers, 8 KV heads, 128 head dimension, FP16) to 200 concurrent users on a single 80 GB GPU. The model weights take about 14 GB. Use the formula to see why raw weights plus KV memory are insufficient for an admission decision.

The raw calculation leaves only a narrow margin. Whether 200 active sequences fit depends on measured activation, workspace, allocator, and fragmentation headroom for the actual serving engine; do not turn an unmeasured reserve into a promised concurrency count.

---

Dynamic token budgeting

In production environments, context length isn't a static limit determined solely by the model's architecture. Instead, it's a dynamic memory budget that dictates how many concurrent users your system can support. Every additional token of context required by one user reduces the available GPU memory (VRAM, Video RAM) for everyone else.

To serve models at scale, inference engines have to enforce these budgets strictly. When a request comes in, the system checks the available GPU memory. If the required KV cache for the new request (plus existing ones) exceeds the remaining capacity, the request waits in a queue. That's why schedulers track memory pressure right alongside latency metrics.

To calculate the maximum affordable context length, we can write a simple capacity planning function. The function takes the total GPU memory, the model's static weight footprint, and its architectural parameters (layers, KV heads, and dimension) as inputs. It computes the available memory per user and divides it by the per-token KV cache size, returning the maximum number of tokens each user can generate:

This calculation drives critical deployment decisions. If you need to support 100 concurrent users but only have the budget for 5,000 tokens each, you might need to add another GPU node, reduce the model precision using quantization, or implement stricter context window limits at the application layer. Treat the result as an upper bound, not a safe production limit: you still need headroom for activations, communication buffers, allocator slack, and the serving runtime itself.

Common mistake: Using the query head count instead of the KV head count. A 70B-class model might have 64 query heads but only 8 KV heads thanks to GQA. If you plug 64 into the formula, you get an 8× memory overestimate and a pessimistic concurrency plan. Check the model card for num_key_value_heads, not num_attention_heads.

An online admission check can reserve KV capacity using each request's prompt plus output budget rather than admitting every request at the maximum architectural context:

---

Production optimizations

Once you understand the two-phase bottleneck, the next question is how production systems work around it. Engineering teams use three major techniques to smooth the tradeoffs between prefill and decode and maximize hardware utilization.

Chunked prefill

On shared serving hardware, large prefills can delay decode operations, creating a TTFT-TPS tradeoff: prioritizing new prefills can stall existing decode streams.

Chunked prefill splits long prompts into smaller chunks, interleaving them with decode steps:

Analogy (factory assembly line): Without chunked prefill, it's like shutting down the entire factory assembly line to set up for a new product. All existing products stop moving while you retool. With chunked prefill, you retool one station at a time while the rest of the line keeps running. Existing requests keep flowing (decode continues) while the new request is gradually set up (prefilled in chunks).

The timeline below illustrates how chunked prefill avoids stalling decodes by breaking up the massive prefill block. By interleaving smaller prefill chunks with ongoing decode steps, the system maintains a steady flow of output tokens for existing users while gradually processing the new prompt:

Benefits

Chunked scheduling can improve GPU utilization by mixing compute-heavy prefill with memory-heavy decode, while protecting streaming cadence and tail latency. vLLM documents chunked prefill as a scheduling optimization,^[9] and systems like Sarathi-Serve^[10] study it explicitly.

Prefill-decode disaggregation

In a standard setup, a single GPU handles both prefill and decode phases for its assigned requests. However, because prefill is compute-bound and decode is memory-bandwidth bound, using the same hardware for both can create head-of-line blocking and muddle hardware sizing. Modern systems (Splitwise^[11], DistServe^[12], Mooncake^[13]) explore separating prefill and decode onto different GPU pools when that isolation benefit outweighs the KV-transfer cost:

Architectural benefits

• Less cross-phase interference: prefill bursts are less likely to stall decode
• Independent scaling: add prefill GPUs for prompt-heavy workloads, decode GPUs for concurrent users
• Hardware matching: use compute-optimized GPUs for prefill, high-bandwidth GPUs for decode

You do pay for moving KV state across the interconnect, so disaggregation is most attractive when prompts are long, traffic is bursty, or TTFT/ITL isolation matters more than the extra transfer overhead.

KV cache quantization

Store KV cache in an 8-bit format such as FP8 instead of FP16/BF16 to roughly halve the cache footprint. Research systems have demonstrated sub-8-bit KV cache quantization, including 3-bit and 2-bit methods, with model-specific quality evaluation required before deployment.^[14][15] vLLM documents FP8 KV-cache support and scaling configuration; support and accuracy trade-offs depend on the engine and hardware.^[16] See our model quantization deep-dive for the techniques behind weight and activation quantization:

$\text{KV Cache (8-bit)} = \frac{\text{KV Cache (FP16/BF16)}}{2}$

Reading the formula: an 8-bit K/V tensor payload uses 1 byte per value instead of 2 bytes (FP16/BF16), so its raw payload is halved for the same sequence length and concurrency. Scaling metadata and runtime buffers mean allocated memory savings may differ slightly.

While quantizing weights reduces the static memory footprint of the model, quantizing the KV cache specifically attacks the dynamic memory bottleneck that limits concurrency. Some serving engines now support KV cache quantization directly. If KV memory is the dominant constraint, moving from 16-bit to 8-bit caching can come close to doubling concurrency on the same hardware. In practice, the gain is smaller once you account for model weights, allocator overhead, and other runtime buffers.

---

Mastery check

Key concepts

• Prefill vs decode as two separate inference phases
• TTFT, TPS, inter-token latency (ITL), and time per output token (TPOT)
• Arithmetic intensity and why the bottleneck flips after token 1
• KV-cache memory formula and why GQA changes the head count
• Context length as a concurrency budget, not only a model-card limit
• Chunked prefill, prefill-decode disaggregation, and KV-cache quantization

Evaluation rubric

• Foundational: Explains why TTFT ends at token 1 and why decode remains sequential after that.
• Intermediate: Diagnoses whether a user complaint points to prefill latency or decode pacing.
• Intermediate: Derives the KV-cache formula and uses num_key_value_heads rather than the full query-head count.
• Advanced: Explains why prefill is usually compute-bound while single-stream decode is usually memory-bandwidth-bound.
• Advanced: Estimates whether a deployment fits by combining model weights, KV cache, and runtime headroom.
• Advanced: Chooses among chunked prefill, disaggregation, or KV-cache quantization based on the real bottleneck.

Follow-up questions

Common pitfalls

"LLMs generate tokens one at a time"

Symptom: You describe the whole request as sequential and then cannot explain the large pause before token 1. Cause: Decode is sequential, but prefill processes the full prompt in parallel before streaming begins. Fix: Split the request into two phases every time you reason about latency: prefill for first-token delay, decode for streaming cadence.

"More GPUs always means faster generation"

Symptom: A team adds more GPUs and expects single-request TPS to scale linearly. Cause: Single-stream decode is often limited by memory bandwidth and communication overhead, not raw compute alone. Fix: Ask what bottleneck you are relieving. If decode is bandwidth-bound, look at HBM bandwidth, scheduling, quantization, or a different parallelism strategy before adding more devices.

"A 70B BF16 throughput estimate uses one H100-80GB"

Symptom: A bandwidth calculation divides one H100's bandwidth by a 140 GB weight tensor. Cause: The estimate ignores capacity: 140 GB of weights cannot reside on one 80 GB device. Fix: Choose a feasible parallel layout first, then estimate using each device's shard and account for interconnect overhead in benchmarks.

"Context length is only a model limitation"

Symptom: Product plans assume every user can use the model's full architectural context without affecting concurrency. Cause: The architectural limit and the production memory budget were treated as the same thing. Fix: Turn context policy into a capacity calculation. Budget KV state per user, then cap prompt and output lengths to preserve concurrency headroom.

"TTFT and TPS improve together"

Symptom: The system optimizes first-token latency aggressively, but active streams become choppy. Cause: Large prefills and smooth decode compete for the same GPU time, so improving one can hurt the other. Fix: Measure TTFT and decode metrics separately. Then tune scheduling, such as chunked prefill, around the actual SLO you need to protect.

"Using query heads for KV-cache sizing"

Symptom: Capacity plans are off by a large factor and the service either looks impossibly expensive or crashes under load. Cause: The formula used num_attention_heads instead of num_key_value_heads on a GQA model. Fix: Check the model config directly. KV memory uses the stored K/V head count, not the full query-head count.

---

Bringing it together

You now have a concrete mental model for how LLM inference behaves in production. When a request arrives, the engine executes a usually compute-heavy prefill pass over the prompt. That prefill drives model-side TTFT. Then the system falls into an often memory-bandwidth-bound low-batch decode loop that generates one token at a time and drives streamed TPS. The KV cache is the bridge between those phases, and it grows with every token, which is why remaining memory capacity, not only compute, often limits concurrency.

If you can explain why a long prompt hurts TTFT more than TPS, why doubling decode speed requires more HBM bandwidth rather than more TFLOPS, and how to estimate whether 100 concurrent users fit on your GPU cluster, you are already ahead of most candidates in an AI infrastructure interview.