Inference: TTFT, TPS & KV Cache

At 09:17, an on-call engineer sees a familiar support report: the agent found the answer, but the chat window stays blank, then the reply arrives in uneven bursts. The GPU dashboard says its compute rate is healthy. Both observations can be true. The blank interval and the later stream are different pieces of one model call.

Trace that call from request arrival. Queueing and tokenization belong to its initial path. Then prefill runs the model over the prompt and builds reusable attention state. After the first token, decode repeats a one-token step until the response ends. Call that reusable state the key-value (KV) cache; it saves recomputation but consumes GPU memory.

Three counters make the trace operational: time to first token (TTFT) for the initial wait, tokens per second (TPS) for later output, and cache occupancy for the prompt plus generated tokens. A longer conversation can slow decode because full-attention steps read a longer cached prefix while still rereading the model weights.

Prefill versus decode: Autoregressive generation has two phases, and they often hit different hardware limits. That split is the reason later serving lessons exist, from KV cache paging to continuous batching.

Why doesn't a chat response appear all at once after you submit a prompt?

Answer

The model first runs a prefill pass over the whole prompt to build prompt state and produce the first token. After that, decode generates one token at a time because each new token depends on the tokens already produced.

---

The two phases of LLM inference

Use an unchunked request as the baseline. Prefill consumes prompt tokens together and writes their K/V state. Decode then consumes one new token per iteration, reads the cached prefix, predicts the next distribution, and appends one more K/V entry.

The handoff is sequential even though the prompt work is parallel:

What is the cleanest mental split between prefill and decode?

Answer

In an unchunked baseline, prefill processes prompt tokens in parallel and builds the initial KV cache, so it drives time to first token. Decode processes one new token per step while reading model weights and cached K/V state, so it drives streaming speed and inter-token latency.

Phase 1: Prefill (processing the prompt)

In the baseline, all input tokens enter one forward pass in parallel. Each weight matrix can serve many prompt-token rows, giving the GPU large matrix multiplies to keep busy. For long prompts on modern accelerators, that shape is often compute-bound, although batch shape, kernel implementation, and hardware can move the boundary. Chunked prefill changes this schedule later for latency reasons. A toy trace looks like this:

The wall-clock interval from request arrival to the first output token is TTFT. People feel that wait at the chat box, so the target belongs to the interaction, not to a generic model benchmark. Choose it from measured user tolerance and include the queueing and network path your product actually exposes.

Phase 2: Decode (generating output tokens)

Once token 1 is sampled, the prompt needn't be processed again. The model generates subsequent tokens one at a time, autoregressively: feed the newest token through the model, reuse cached K/V for the prefix, append its K/V, and produce the next distribution. The trace below shows the loop:

At low batch, each step does little matrix work for a large byte movement. The GPU reads model weights and the request's KV cache from GPU high-bandwidth memory (HBM), so the arithmetic units can wait on data instead of doing math. In a full-attention layer, a longer response also means a longer cached prefix to read. Per-token latency can rise even though the weight tensor is unchanged.

Read the following timeline as instrumentation, not as a model benchmark. Arrival is 0 ms; four output tokens are observed at 320, 355, 392, and 428 ms. The first timestamp closes TTFT. Only the gaps after it describe decode cadence.

Why prefill and decode hit different roofs

The useful lens is arithmetic intensity: FLOPs performed per byte fetched from high-bandwidth memory (HBM). It asks whether a phase has enough reuse to keep math units busy.

During prefill, one weight matrix serves many prompt-token rows. That reuse raises FLOPs per byte.

Qwen3.6-27B gives a concrete scale: its 27B-parameter BF16 language weights occupy about 54 GB ($27 \times 10^{9} \times 2$ bytes), before KV cache, runtime buffers, or vision-encoder tensors.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} A long prompt can amortize that footprint across many matrix multiplications, so Tensor Cores often become the limiting resource. Short prompts or an underfilled batch can behave differently.

During low-batch decode, one new token supplies only a thin matrix-vector-shaped workload. Each step still moves weights and cached K/V, but there are fewer FLOPs to amortize those bytes. HBM traffic can therefore set token cadence before peak compute does.

Increasing decode batch can change that balance by letting one weight read serve multiple active sequences.

The roofline model^{[2]Reference 2Roofline: An Insightful Visual Performance Model for Multicore Architectureshttps://doi.org/10.1145/1498765.1498785} turns that lens into a ceiling: at intensity $I$, attainable throughput is bounded by $\min(P_{compute},\; I \times BW_{HBM})$. The two roofs meet at a ridge point. Below it, bandwidth is the candidate limit; above it, compute is the candidate limit.

Treat that curve as a ceiling, not proof that a kernel reaches either roof.

For scale, NVIDIA lists 80 GB of HBM and 3.35 TB/s peak HBM bandwidth for H100 SXM.^{[3]Reference 3H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/} Those are hardware specifications, not sustained application measurements.

Qwen3.6-27B's roughly 54 GB BF16 language weights fit in that capacity on paper, but its official 262,144-token native context still leaves KV-cache policy and runtime headroom to budget.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}

That distinction motivates IO-aware kernels such as FlashAttention^{[4]Reference 4FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.https://arxiv.org/abs/2205.14135} for long prefills, while PagedAttention^{[5]Reference 5Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} addresses KV-cache allocation and reuse during serving.

Put a measured point on the roofline

Use one kernel or one clearly defined phase at a time. Record elapsed time, FLOPs, and bytes moved through HBM for a correctness-checked workload. Then compute $P_{measured}=\text{FLOPs}/\text{time}$ and $I=\text{FLOPs}/\text{HBM bytes}$, and plot $(I, P_{measured})$ against roofs for the same GPU, precision, and kernel scope. Allocated cache capacity is not the same as bytes actually moved, so use profiler traffic where possible.

Near the sloped roof with HBM traffic near an empirically sustained bandwidth ceiling, bandwidth is a plausible limit. Near the flat, precision-specific compute roof with unsaturated HBM, compute is more plausible. A point far below both roofs calls for checking launch overhead, underfilled work, synchronization, communication, or accounting before changing model architecture.

To find which input moved the point, hold engine, hardware, precision, and correctness constant while varying one knob:

• Batch: Raising decode batch can reuse weights across sequences and move intensity right, but may worsen per-request ITL.
• Sequence: A longer prompt changes prefill work. A longer full-attention decode sequence adds K/V reads and can move intensity left.
• Bytes: Changing weight or KV precision changes traffic and often the compute roof too. Remeasure and redraw both axes instead of comparing points from different precisions.

Why is prefill usually compute-bound while single-stream decode is usually memory-bandwidth-bound?

Answer

Prefill reuses loaded weights across many prompt tokens in parallel, producing many FLOPs per byte fetched. Decode handles one new token at a time, so it repeatedly reads model weights and the growing KV cache for relatively thin matrix-vector work.

---

Four numbers for one request

The alert from the opening has two clocks. TTFT stops when token 1 appears; decode metrics start with the gaps that follow. Keep them separate before diagnosing a slow request.

TTFT (time to first token)

TTFT is wall-clock time until the first output token appears.^{[6]Reference 6Metricshttps://docs.vllm.ai/en/stable/design/metrics/} At kernel level, prefill is usually largest component. User-visible TTFT also includes tokenization, queueing, scheduling, and network overhead. Some engines start their timer when a request is scheduled and report queue time separately, so check definitions before comparing series.

Once queueing is controlled, longer prompts generally add prefill work, but exact scaling depends on prompt batch, attention implementation, and hardware. Set TTFT SLO from your product interaction, not from a headline benchmark.

TPS (tokens per second)

Here, TPS means decode throughput: how fast later output tokens arrive after token 1. For one request, use $1000 / \text{mean ITL}_{ms}$. Some dashboards divide all output tokens by end-to-end request time instead, folding prefill into the rate and making a long prompt look like a slow generator.

At low batch, HBM bandwidth often limits decode. Architecture, quantization, parallelism, engine, context length, and batch shape all change the result, so measure the deployed configuration. Production engines use batched inference, often with continuous batching,^{[7]Reference 7Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu} to raise aggregate throughput across requests.

ITL (inter-token latency)

ITL is the gap between consecutive output tokens.^{[6]Reference 6Metricshttps://docs.vllm.ai/en/stable/design/metrics/} For one isolated request, ITL in seconds is the inverse of TPS ($ITL = 1/TPS$). For millisecond traces, use $TPS = 1000/ITL_{ms}$.

Higher ITL makes a stream feel choppier. Under batching, scheduling and resource contention can widen individual gaps even when aggregate TPS improves. Alert on distributions from product traffic, not one copied threshold.

TPOT (time per output token)

TPOT averages time per generated output token after token 1. It may be the mean of one request's ITL values or an aggregate statistic across requests, depending on the benchmark definition.^{[6]Reference 6Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

Read both numbers together. ITL exposes jitter and stalls; TPOT summarizes decode pacing. A prefill interruption or scheduling decision can worsen either one without changing model weights.

The four metrics are summarized below:

If users complain that nothing appears for a long time, which metric do you inspect first? What if streaming looks choppy after it starts?

Answer

Inspect TTFT first when the initial pause is too long, because that points to prefill, queueing, tokenization, or scheduling before the first token. Inspect ITL, TPOT, and TPS when streaming is choppy after generation begins, because those describe decode pacing and system contention.

This tiny router shows how an alert can choose an investigation path. Its thresholds are illustrative policy inputs, not benchmark results:

Back-of-the-envelope: a bandwidth upper bound

Before running a serving benchmark, estimate one optimistic limit. For a single low-batch decode stream, most model weights must be read for each token, so memory bandwidth is a useful first bound:

$\text{Max TPS} \approx \frac{\text{HBM Bandwidth}}{\text{Model size (bytes)}}$

For this calculation, assume dense Qwen3.6-27B language weights in BF16, one H100 SXM, weights resident on that GPU, one low-batch stream, and ideal peak HBM bandwidth. The model card implies about 54 GB of language weights, before KV cache, runtime buffers, extra tensors, and allocator headroom.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}

NVIDIA lists 3.35 TB/s peak HBM bandwidth for H100 SXM.^{[3]Reference 3H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/} Treat that specification as an optimistic roof, not a sustained application measurement.

$\text{Ideal weight-read bound} \approx \frac{3{,}350 \text{ GB/s}}{54 \text{ GB}} \approx 62 \text{ tokens/sec}$

Treat 62 tok/s as a theoretical peak-HBM ceiling, not a measured single-stream result. Two refinements keep this estimate honest:

1. Sustained bandwidth must come from a benchmark on selected hardware and kernel mix. Replace 3,350 GB/s with that measured value; skip any universal peak-to-sustained percentage.
2. KV-prefix traffic grows with sequence length $s$: each full-attention decode step also reads prior K/V for that request and any batch mates. As $s$ rises, weight streaming is no longer the only HBM term, so the effective ceiling falls below this pure weight-read bound.

The bound omits activation memory, kernel overhead, scheduling, and interconnect traffic when the model is sharded. Benchmark selected engine, parallelism layout, model precision, prompt/output distribution, and concurrency to get real TPS. Batched inference can improve aggregate throughput by sharing weight reads, but it changes arithmetic intensity and per-request ITL at the same time.

Research note: For low-batch decode, quantization can move the weight-streaming bound by reducing bytes read per token. PagedAttention^{[5]Reference 5Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} improves KV-cache packing and sharing, while continuous batching^{[7]Reference 7Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu} improves utilization across requests. These techniques address related serving constraints, but they aren't interchangeable.

Why does the rough bandwidth estimate divide HBM bandwidth by model size?

Answer

For a low-batch decode stream, each token step must effectively move the weight footprint through GPU memory. Dividing ideal HBM bandwidth by the weight footprint gives an optimistic upper bound before KV-cache reads, kernels, communication, and runtime overhead.

---

The KV cache: the dynamic capacity bottleneck

The 62 tok/s calculation asked how fast bytes might move. Capacity asks a different question: after weights and runtime buffers are resident, how much HBM remains for active request state? Often, KV cache consumes most of it.

Cached keys and values

During attention, each eligible layer computes Key and Value projections for each token. If a decode step discarded those tensors, it would have to process the whole prefix again to recreate them. That repeated work grows with every generated token.

What the cache is: Reusable attention state for tokens already seen. For each processed token, the model stores key routing vectors (K) and value content vectors (V). When the next token attends to earlier context, it reads those tensors instead of re-deriving the whole prefix. The cache grows with each token, and its size determines how many concurrent sequences fit in GPU memory.

The trace below shows how the cache expands with each generated token:

Growing state creates an allocator problem as well as a byte-count problem. Systems such as vLLM use PagedAttention^{[5]Reference 5Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} to divide KV storage into fixed-size blocks (pages), much like operating-system virtual memory. A live sequence can then occupy non-contiguous blocks instead of reserving one maximum-length span. Paging reduces fragmentation and over-reservation; partially filled tail blocks and metadata overhead still remain.

What does the KV cache save, and what does it cost?

Answer

It saves compute by avoiding recomputation of old key and value projections at every decode step. It costs GPU memory that grows with layers, KV heads, head dimension, sequence length, precision, and active requests.

KV cache memory formula

For one sequence with a standard growing K/V cache:

$\text{KV Cache} = 2 \times L \times n_{kv} \times d_h \times s \times p$

Read it left to right: for every layer that stores K/V, every KV head, and every position, store one Key vector and one Value vector (the leading $2$), each of dimension $d_h$ and element size $p$. For $B$ concurrent sequences, multiply by $B$. The linear dependence on $s$ and $B$ is why a cache budget disappears quickly at long context or high concurrency.

Where:

• $L$ = number of layers that store growing K/V (not every layer in a hybrid model)
• $n_{kv}$ = number of KV heads (reduced with GQA/MQA^{[8]Reference 8GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.https://arxiv.org/abs/2305.13245})
• $d_h$ = head dimension
• $s$ = sequence length
• $p$ = bytes per element (2 for FP16/BF16, 1 for an 8-bit cache such as INT8 or FP8)

Concrete example: Qwen3.6-27B-style GQA sizing

$$\begin{aligned} \text{KV Cache} &= 2 \times 16 \times 4 \times 256 \times 4096 \times 2 \\ &\approx \textbf{0.27 GB} \; (\approx \textbf{0.25 GiB}) \textbf{ per sequence} \end{aligned}$$

The arithmetic is $2$ (K and V) × 16 attention layers × 4 KV heads × 256 values per head × 4,096 positions × 2 bytes. Qwen3.6-27B's official configuration has 64 layers arranged as 16 repeats of three Gated DeltaNet layers plus one Gated Attention layer.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Only those 16 attention layers belong in this standard growing-KV calculation. The 48 DeltaNet layers have separate recurrent state, which must be budgeted separately. This example sizes language-side attention state, not vision-encoder memory.

With GQA, this cache is 6× smaller than storing separate K/V tensors for all 24 query heads in those 16 layers. The same geometry with 24 KV heads would need about 1.61 GB (1.5 GiB) per sequence.

Production note: A weight-fit check is not an admission policy. Qwen3.6-27B's BF16 language weights are about 54 GB, so they fit on one H100-80GB on paper, but remaining HBM must also cover KV cache, DeltaNet state, runtime buffers, prefix cache, optional vision-encoder weights, and allocator overhead. Its 262,144-token native window is usable only when policy reserves enough memory for active prompt-plus-output tokens.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}

Try it yourself: A colleague says a 7B model (32 layers, 8 KV heads, 128 head dimension, FP16) can serve 200 concurrent users on one 80 GB GPU. The weights take about 14 GB. Use the formula, then leave a runtime reserve, to test that claim.

Raw weights plus KV leave a narrow margin. Whether 200 active sequences fit depends on measured activations, workspace, allocator behavior, and fragmentation for the serving engine. The True result under one arbitrary reserve is not a concurrency promise.

Why must the KV-cache formula use num_key_value_heads instead of num_attention_heads for GQA models?

Answer

GQA shares one K/V head across multiple query heads. The cache stores keys and values, not every query head, so using the full query-head count overestimates cache memory by the query-to-KV grouping factor.

---

Dynamic token budgeting

Context length is an architecture limit, not a concurrency budget. The deployable budget is leftover GPU memory after weights and runtime buffers, divided across active requests. One extra reserved token for request A leaves less room for request B.

When a request arrives, the engine estimates its prompt-plus-output KV demand. If that demand plus live cache exceeds remaining capacity, the scheduler must queue, reject, or preempt it. Capacity and latency belong on the same dashboard because overcommit is not a recovery plan.

The function below turns that policy into an upper bound. It takes GPU memory, static weight footprint, runtime reserve, cache shape (layers, KV heads, head dimension), and concurrency, then returns prompt-plus-output tokens per user:

This calculation gives a planning limit, not a safe production limit. If 100 users need 5,000 reserved tokens each and the budget falls short, options include additional memory, lower precision, fewer active tokens, or application caps. Validate any choice with activations, communication buffers, allocator slack, DeltaNet state, and serving-runtime measurements.

Common mistake: Plugging query heads into a KV formula. Qwen3.6-27B lists 24 query heads but 4 KV heads for its Gated Attention path.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Using 24 overestimates this cache by 6×. Read num_key_value_heads and the model's layer layout; using all 64 layers would also count 48 DeltaNet layers outside this growing-KV formula.

An online admission check can reserve capacity from each request's prompt-plus-output token budget instead of assuming every request consumes the maximum architectural context:

Why is maximum context length a production policy, not a model-card number alone?

Answer

Every extra active token consumes KV-cache memory. Allowing every user to use the model's full architectural context can collapse concurrency, so production systems cap prompt length, output length, and active tokens based on GPU memory budgets and latency goals.

---

Stop a new prefill from stalling live streams

Capacity and scheduling solve different failures. A request can fit in HBM and still make every active stream wait if its long prefill monopolizes a shared worker. The next choices either bound that interference or reduce per-request cache bytes.

Chunked prefill

On shared hardware, a large prefill competes with decode for the same iteration budget. Prioritizing it can improve the new request's TTFT while stalling existing streams, creating a TTFT-TPS tradeoff.

Chunked prefill makes that tradeoff explicit. Split a long prompt into smaller chunks, run one bounded slice, serve active decode steps, then continue the prompt. Without chunking, a 10K-token prefill can occupy the iteration budget until completion. With 2K-token chunks, active streams get turns between slices while the new request advances toward token 1:

Why this helps

Mixing compute-heavy prefill slices with memory-heavy decode can protect streaming cadence and tail latency. Chunk size is an operating point: smaller slices give decodes more frequent turns but can add scheduling overhead or delay the new request. vLLM documents this control,^{[9]Reference 9Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html} and Sarathi studies piggybacking decodes on chunked prefills.^{[10]Reference 10Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.https://arxiv.org/abs/2308.16369} Measure p95/p99 ITL and TTFT together.

Why does chunked prefill improve tail latency for existing decode streams?

Answer

A long prefill is compute-heavy and can monopolize the GPU. Chunking breaks that work into smaller pieces, letting the scheduler interleave decode steps so active users keep receiving tokens instead of waiting behind one giant prompt.

Prefill-decode disaggregation

Chunking still makes one GPU serve two competing phases. If prompt bursts remain the cause of tail ITL, place prefill and decode on separate GPU pools. Long-prompt prefill is often compute-heavy; low-batch decode is often bandwidth-heavy. Splitwise,^{[11]Reference 11Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677} DistServe,^{[12]Reference 12DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670} and Mooncake^{[13]Reference 13Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079} study this separation when its isolation benefit outweighs KV-transfer cost:

What changes

Separate pools remove much of the head-of-line interference: a prefill burst no longer consumes decode iterations on same worker. They also let capacity follow workload shape, adding prefill workers for prompt bursts and decode workers for active sequences. Finally, each pool can use a topology that suits its phase rather than forcing one compromise.

The cost is moving initial KV state across an interconnect before decode starts. Disaggregation is attractive only when measured transfer time and operational complexity are smaller than interference it removes. Long prompts, bursty arrivals, and strict TTFT/ITL isolation are signals to test it, not guarantees.

The comparison below uses illustrative measurements. Replace them with paired runs from same hardware, engine, precision, workload, and correctness check before making a deployment choice.

When is prefill-decode disaggregation worth its extra KV-transfer cost?

Answer

It's most useful when prefill bursts hurt streaming latency or when prompt length and active-user load scale differently. Separate GPU pools let you size compute-heavy prefill and memory-bandwidth-heavy decode independently, but the KV transfer must be cheaper than the interference it removes.

KV cache quantization

If cache capacity, rather than latency interference, is the failing resource, reduce bytes per cached value. An 8-bit format such as FP8 uses about half the raw payload of FP16/BF16. Research systems also explore 3-bit and 2-bit caches, but quality and kernel support are model-, hardware-, and engine-specific.^{[14]Reference 14KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationhttps://arxiv.org/abs/2401.18079[15]Reference 15KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachehttps://arxiv.org/abs/2402.02750} vLLM documents FP8 KV-cache configuration and calibration.^{[16]Reference 16Quantized KV Cachehttps://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/} See our model quantization deep-dive for weight and activation trade-offs:

$\text{KV Cache (8-bit)} = \frac{\text{KV Cache (FP16/BF16)}}{2}$

An 8-bit K/V payload uses 1 byte per value instead of 2, so raw cache bytes are halved for the same sequence length and concurrency. Scaling metadata, padding, and runtime buffers mean allocated memory savings can be smaller.

Weight quantization shrinks static model memory. KV quantization targets dynamic per-request state, which grows with context and concurrency. If KV is the dominant resource and all other reservations stay fixed, halving raw bytes can approach a 2× cache-only concurrency gain. Real gain is lower when weights, allocator overhead, and runtime buffers consume most HBM.

The following planning example reuses Qwen3.6-27B-style GQA dimensions. Each active user reserves 5,120 prompt-plus-output tokens of FP16 KV state. The budget is the 18 GB left after 54 GB of language weights and an 8 GB runtime reserve on one 80 GB GPU, not the whole card:

Why can KV-cache quantization improve concurrency even if model weights are already quantized?

Answer

Weight quantization shrinks the static model footprint. KV-cache quantization shrinks per-token, per-request state, which is the dynamic memory cost that grows with context length and concurrent users.

---

One request, two bottlenecks

A request crosses two different operating regimes. Prefill turns prompt tokens into logits and initial K/V state, often making compute the first limit on TTFT. Decode then emits one token per step, often making HBM traffic the first limit on streaming cadence. The KV cache connects them and grows with every active token, so capacity can become the first limit on concurrency.

The useful diagnosis is causal: a long prompt expands prefill work and TTFT; a low-batch decode stream repeatedly moves weights and cached state; a longer context adds K/V reads; a larger batch can trade per-request ITL for aggregate reuse. Those are hypotheses to test with measurements, not labels to copy from a model card.

Build a serving budget

For a real serving budget, record hardware and engine, model and precision, prompt/output distribution, arrival process, concurrency, correctness result, and measured HBM/compute counters. Then report weight memory, KV bytes per token, TTFT and ITL p50/p99, raw throughput, and goodput under the product SLO. The first release constraint should be visible as compute, bandwidth, capacity, or an unmodeled overhead.

Build a serving-budget table from one measured workload. Record the inputs, calculate weight and KV reservations, compare TTFT and ITL with the SLO, and trace the first constraint to compute, bandwidth, capacity, or overhead. Keep that table as the release artifact, then rerun it when model, prompt mix, precision, or concurrency changes.