Inference: TTFT, TPS & KV Cache

When you send a prompt to ChatGPT and see the response appear word by word, you're watching inference: the process of a trained model generating output. But why does it stream one word at a time? Why is the first word slow and the rest faster? And why does a longer conversation make everything slower? Understanding these mechanics (the two-phase process, the memory bottleneck, and the key performance metrics) is essential for anyone building or optimizing production LLM systems.

💡 Key insight: LLM inference has two distinct phases (prefill and decode) with fundamentally different hardware bottlenecks. Knowing which phase is the bottleneck, and why, is the foundation for every optimization in this space, from KV cache management to continuous batching.

---

The two phases of LLM inference

Every LLM request goes through two distinct computational phases with fundamentally different hardware bottlenecks:

🍽️ Analogy, restaurant kitchen: Prefill is like a chef reading the entire recipe at once, gathering all ingredients, and prepping the mise en place. It's intense upfront work but highly parallelizable (multiple sous chefs can chop simultaneously). Decode is like plating one dish at a time in a specific order: each plate (token) must wait for the previous one, and the bottleneck is how fast you can carry ingredients from the fridge (memory bandwidth), not how fast you can cook (compute).

Phase 1: Prefill (processing the prompt)

The model processes your entire input prompt in parallel. Every token is attended to simultaneously in a single forward pass. This is compute-bound, limited by GPU FLOPS, not memory bandwidth. For example, when starting generation, the prefill phase processes the input text to produce the first token:

text
1Input: "Explain quantum computing in simple terms"
2→ 6 tokens processed simultaneously
3→ Produces KV cache entries for all 6 tokens
4→ Produces logits (unnormalized probability scores) for the FIRST output token

The time from request to the first output token is called TTFT (Time to First Token). For a user staring at a chat interface, this is the most noticeable latency.

Phase 2: Decode (generating output tokens)

After producing the first token, the model generates subsequent tokens one at a time, autoregressively. Each new token requires a forward pass through the entire model, but only processes that single new token (reusing cached K/V from all previous tokens). This step-by-step process builds the response iteratively:

text
1Step 1: "Quantum" → add to KV cache → forward pass → "computing"
2Step 2: "computing" → add to KV cache → forward pass → "is"
3Step 3: "is" → add to KV cache → forward pass → "like"
4...

This phase is memory-bandwidth bound, not compute-bound. The bottleneck is reading the model weights and KV cache from GPU HBM (High Bandwidth Memory) for each token. The matrix multiplications are small (batch=1), so the GPU's arithmetic units are mostly idle, waiting for data to arrive.

The arithmetic intensity explanation

The key difference is arithmetic intensity (FLOPs per byte loaded from memory):

On an H100 GPU: ~990 TFLOPS (Tera Floating Point Operations Per Second) compute, ~3.35 TB/s HBM bandwidth. During decode, the GPU is reading ~140GB of model weights to process a single token. The computation itself takes microseconds, but loading the weights takes milliseconds. This memory-bound nature is what motivates architectural changes like FlashAttention^[1] to minimize HBM access.

---

Key performance metrics

TTFT (Time to First Token)

• •Measures prefill latency, how quickly the model starts responding
• •Dominated by prompt length and model size
• •Scales roughly linearly with prompt token count (for long prompts)
• •Critical for: interactive applications, voice assistants, code completion

TPS (Tokens Per Second), also called "decode throughput"

• •Measures how fast output tokens are generated after the first
• •Single request: typically 30–80 TPS for large models on high-end GPUs
• •Batched inference (often using continuous batching^[2]): hundreds or thousands of aggregate TPS
• •Determined by: memory bandwidth, not compute

ITL (Inter-Token Latency)

• •Time between consecutive output tokens: ITL = 1/TPS for a single request
• •Users perceive stuttering when ITL exceeds ~100ms
• •Under batching, ITL increases as more requests share the GPU

TPOT (Time Per Output Token)

• •System-level metric accounting for batching overhead and scheduling
• •TPOT ≥ ITL due to scheduling delays and contention

---

The KV cache: the dominant bottleneck

What is the KV cache?

During attention, each layer computes Key and Value projections for every token. Without caching, generating token $N$ would require recomputing attention over all $N-1$ previous tokens from scratch, which is quadratic in sequence length.

📋 Analogy, exam reference sheet: The KV cache is like a reference sheet you build during an exam. For each question (token) you've already answered, you write down the key facts (K) and your reasoning (V) on the sheet. When the next question references a previous one, you glance at your reference sheet instead of re-deriving everything from scratch. The sheet grows with each question, and its size determines how many questions you can handle before running out of paper (GPU memory).

The KV cache stores these K and V tensors. As the sequence grows, the KV cache accumulates data for each token to avoid redundant calculations:

text
1Token 1: Compute K₁, V₁ → Store in cache
2Token 2: Compute K₂, V₂ → Store; Attend to [K₁,K₂], [V₁,V₂]
3Token 3: Compute K₃, V₃ → Store; Attend to [K₁,K₂,K₃], [V₁,V₂,V₃]
4...

KV cache memory formula

For a single sequence:

$\text{KV Cache} = 2 \times L \times n_{kv} \times d_h \times s \times b$

Reading the formula: for every layer ($L$), every KV head ($n_{kv}$), every position in the sequence ($s$), we store a Key vector and a Value vector (the "2") of dimension $d_h$, each taking $b$ bytes. Multiply it all together and this cache can easily reach gigabytes for long sequences.

Where:

• •$L$ = number of layers
• •$n_{kv}$ = number of KV heads (reduced with GQA/MQA (Grouped-Query/Multi-Query Attention)^[3])
• •$d_h$ = head dimension
• •$s$ = sequence length
• •$b$ = bytes per element (2 for FP16, 1 for INT8)

Concrete Example: Llama 3.1 70B

$\text{KV Cache} = 2 \times 80 \times 8 \times 128 \times 4096 \times 2 = \textbf{1.34 GB per sequence}$

Reading the formula: $2$ is for K and V, $80$ is the number of layers, $8$ is KV heads, $128$ is head dimension, $4096$ is sequence length, and the final $2$ is FP16 bytes per value.

With GQA (8 KV heads instead of 64), this is 8× smaller than it would be with MHA. Without GQA, the same model would need 10.7 GB per sequence.

🎯 Production tip: Serving a 70B model to 100 concurrent users requires precise KV cache budgeting: Model weights (140 GB FP16) + KV cache (1.34 GB × 100 users = 134 GB) = 274 GB total. Feasible on 4× H100-80GB with tensor parallelism (splitting the model across multiple GPUs).

---

Dynamic token budgeting

In production environments, context length is not a static limit determined solely by the model's architecture. Instead, it is a dynamic memory budget that dictates how many concurrent users your system can support. Every additional token of context required by one user reduces the available VRAM for everyone else.

To serve models at scale, inference engines must strictly enforce these budgets. When a request comes in, the system checks the available GPU memory. If the required KV cache for the new request (plus existing ones) exceeds the remaining capacity, the request must wait in a queue. This is why managing the TTFT-TPS tradeoff is so important for overall throughput.

You can dynamically calculate the maximum affordable context length by subtracting the model weights' footprint from total GPU memory, then dividing the remainder by the number of concurrent users and the per-token KV cache size:

python
1def max_context_for_budget(
2    gpu_memory_gb: float,
3    model_memory_gb: float,
4    num_layers: int,
5    num_kv_heads: int,
6    head_dim: int,
7    dtype_bytes: int = 2,  # FP16
8    num_concurrent: int = 1,
9) -> int:
10    """Calculate maximum context length given memory constraints."""
11    available_memory = (gpu_memory_gb - model_memory_gb) * 1e9
12    
13    # Memory per token in KV cache
14    bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * dtype_bytes
15    
16    # Divide by concurrent users
17    budget_per_user = available_memory / num_concurrent
18    
19    return int(budget_per_user / bytes_per_token)
20
21# Example: Llama 3.1 70B on 4×H100-80GB (320GB total)
22max_tokens = max_context_for_budget(
23    gpu_memory_gb=320,
24    model_memory_gb=140,  # FP16 weights
25    num_layers=80,
26    num_kv_heads=8,  # GQA: 8 KV heads (not 64!)
27    head_dim=128,
28    num_concurrent=50,
29)
30# Result: ~11,000 tokens per user with 50 concurrent users

This calculation drives critical deployment decisions. If you need to support 100 concurrent users but only have the budget for 5,000 tokens each, you might need to add another GPU node, reduce the model precision using quantization, or implement stricter context window limits at the application layer.

---

Production optimizations

Chunked prefill

Large prefills block decode operations, causing TTFT-TPS tradeoff: prioritizing new prefills stalls existing decode streams.

Chunked prefill splits long prompts into smaller chunks, interleaving them with decode steps:

🏭 Analogy, factory assembly line: Without chunked prefill, it's like shutting down the entire factory assembly line to set up for a new product. All existing products stop moving while you retool. With chunked prefill, you retool one station at a time while the rest of the line keeps running. Existing requests keep flowing (decode continues) while the new request is gradually set up (prefilled in chunks).

The timeline below illustrates how chunked prefill avoids stalling decodes:

text
1Without chunked prefill:
2  [Prefill 10K tokens ===========================] [Decode...Decode...Decode...]
3  ↑ All decode requests stall during this prefill
4
5With chunked prefill (chunk=2048):
6  [Prefill chunk1][Decode][Prefill chunk2][Decode][Prefill chunk3][Decode]...
7  ↑ Decode requests continue between chunks

Benefits: Better GPU utilization (mixing compute-bound prefill with memory-bound decode), smoother token streaming, reduced tail latency. Enabled by default in vLLM V1 and discussed in the context of efficient memory management systems like PagedAttention^[4].

Prefill-decode disaggregation

Modern systems (Splitwise^[5], DistServe^[6], Mooncake^[7]) separate prefill and decode onto different GPU pools:

Why this works:

• •No cross-phase interference: prefill never stalls decode
• •Independent scaling: add prefill GPUs for prompt-heavy workloads, decode GPUs for concurrent users
• •Hardware matching: use compute-optimized GPUs for prefill, high-bandwidth GPUs for decode

KV cache quantization

Store KV cache in INT8 or even INT4 instead of FP16 to halve/quarter memory with minimal quality loss (see our model quantization deep-dive for the techniques behind weight and activation quantization):

$\text{KV Cache (INT8)} = \frac{\text{KV Cache (FP16)}}{2}$

Reading the formula: INT8 uses 1 byte per value instead of 2 bytes (FP16), so KV memory is cut in half for the same sequence length and concurrency.

This doubles the number of concurrent users you can serve on the same hardware.

---

Common misconceptions

❌ "LLMs generate tokens one at a time" Only true for the decode phase. Prefill processes the entire prompt in parallel.

❌ "More GPUs = faster generation" For a single request, model parallelism adds communication latency. Faster generation requires higher memory bandwidth, not more compute.

❌ "Context length is just a model limitation" In production, context length is a memory budget decision. Shorter contexts = more concurrent users.

❌ "TTFT and TPS improve together" They're often in tension. Optimizing TTFT (prioritizing prefills) hurts TPS (stalling decodes), and vice versa.

---

Key takeaways

1. •Two-phase inference: Prefill (compute-bound, parallel) → Decode (memory-bandwidth-bound, sequential)
2. •TTFT measures prefill speed; TPS measures decode throughput. They're independent and often in tension.
3. •KV Cache is the primary memory bottleneck for concurrent serving; derive the formula from model architecture
4. •GQA/MQA reduces KV cache proportionally to head count reduction (8× for Llama 3.1 70B)
5. •Chunked prefill interleaves prompt processing with decode to avoid stalling
6. •Disaggregation separates prefill and decode onto different GPU pools for independent scaling
7. •Dynamic token budgeting calculates max context from GPU memory, model size, and concurrency

---