Serving Benchmark Engineering

A new serving build finishes 12% more requests per second in a short test. It looks ready until six mixed requests arrive in a tight burst. First tokens slow down, live streams pause, and two responses fail the output schema.

The 12% number wasn't false. It answered the wrong question. A release benchmark must reproduce the workload that creates queueing, measure the moments users feel, and reject fast answers that aren't acceptable answers.

GPU Profiling, Correctness, and Benchmarking established how to time completed GPU work and preserve a kernel receipt. Continuous Batching and Scheduling showed why prompt processing and token generation compete. GPU Serving and Autoscaling turned that contention into a fleet capacity problem, while Distributed Inference Data Plane followed request and KV-cache state across workers. Here those pieces become one release test.

The running artifact is incident-assistant-v3, a fixed request trace for an incident-response assistant. Baseline and candidate must receive the same requests at the same scheduled times. Only then can their operating curves be compared.

Freeze workload before measuring system

A benchmark workload has three independent parts:

1. What arrives: prompt text or token IDs, output policy, priority, shared prefixes, and correctness expectations.
2. When it arrives: a realized timestamp for every request.
3. How completion is judged: latency service-level objectives (SLOs), error rules, and quality checks.

A seed and a distribution name aren't a frozen trace. Library versions can change random-number streams, tokenizers can change token counts, and two candidates can draw different long-tail samples. Generate once, save realized rows, hash file, and replay it.

The small incident-assistant-v3 fixture contains six request identities:

Prompt tokens must be counted with pinned tokenizer. max_tokens is only a cap, not actual output length. Store generated length after run as a result, never as if server knew it at admission.

Forced output lengths have a legitimate but narrower use. Setting an engine to ignore end-of-sequence tokens can isolate scheduler capacity at exact shapes. It also creates requests users wouldn't send and can hide quality changes. Label such run a synthetic stress test, then keep natural-stop trace as release gate.

Preserve distributions, not only averages

An average 512-token prompt can mean every prompt has 512 tokens, or half have 64 and half have 960. Those workloads create different prefill batches, KV-cache pressure, and queue tails.

Keep at least these workload properties:

BurstGPT publishes a real serving trace for studying arrival, concurrency, token-length, conversation, and failure patterns.^{[1]Reference 1BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systemshttps://arxiv.org/abs/2401.17644} It doesn't make those patterns representative of your endpoint. ServeGen builds parameterized, sanitized workloads by composing per-client arrival and data models.^{[2]Reference 2ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productionhttps://www.usenix.org/conference/nsdi26/presentation/xiang-servegen} It remains a model of traffic. Validate either source against your own request classes before turning it into a release fixture.

The fixture now says what and when. Next it needs a load model that doesn't erase overload.

Keep offered load independent of response time

An open-loop generator schedules arrivals independently of server completions. If r04 is due at 760 ms, it arrives then even while r03 is still in prefill. Queue growth remains visible.

A closed-loop generator sends a new request only after a prior response completes. When server slows, generator also slows. That feedback makes a saturated system look calmer because offered rate falls with service rate.

MLPerf Inference makes distinction explicit. Its Server/Interactive scenario uses Poisson arrivals, while Single Stream issues next query after previous one finishes. Valid runs also pair scenario latency rules with quality requirements.^{[3]Reference 3MLPerf Inference Benchmark Rules.https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc} Those rules are useful design evidence. A custom endpoint test isn't an MLPerf result unless it follows full applicable rule set and submission process.

Rate, burstiness, and concurrency are different knobs

Arrival rate $\lambda$ is intended requests per second. Concurrency is number currently outstanding. In a stable system, Little's Law gives useful check:^{[4]Reference 4A Proof for the Queuing Formula: L = lambda Whttps://doi.org/10.1287/opre.9.3.383}

$$L \approx \lambda W$$

$L$ is average outstanding work and $W$ is average time in system. At 2 requests per second and 1.5 seconds average end-to-end latency, expect about 3 requests in flight. A concurrency cap of 2 will throttle generator below intended 2 requests per second.

Burstiness describes variation in inter-arrival gaps. One simple summary is coefficient of variation:

$$CV = \frac{\text{standard deviation of inter-arrival time}}{\text{mean inter-arrival time}}$$

Exponential inter-arrival times from a Poisson process have $CV=1$. A trace with $CV>1$ has more variable gaps by this summary, often with clusters followed by quiet periods. One number still can't preserve diurnal changes, correlated clients, or a planned release burst. Store timestamps when those patterns matter.

Current vllm bench serve can generate finite-rate traffic, vary Gamma-distributed inter-arrival times with --burstiness, cap outstanding requests with --max-concurrency, and report goodput-related metrics.^{[5]Reference 5Benchmark CLIhttps://docs.vllm.ai/en/stable/benchmarking/cli/} Its parameter named burstiness is Gamma shape $k$, not coefficient of variation: $CV=1/\sqrt{k}$, values below 1 produce more bursty gaps, and 1 gives Poisson traffic. Record tool version and realized schedule rather than relying on flag name.

You schedule 3 requests per second, observe 900 ms mean end-to-end latency, and cap client concurrency at 2. Can this run establish capacity at 3 requests per second?

Answer

No. Little's Law predicts about $L=3\times0.9=2.7$ requests in flight at that rate, above client cap. Generator will delay or suppress arrivals before server sees intended load. Raise cap, retain scheduled and sent timestamps, and verify schedule lag before treating run as open-loop capacity evidence.

Detect client-side coordinated omission

Record five timestamps or event series:

If client schedules r04 for 760 ms but can't send until 1,100 ms, measuring from sent_at deletes 340 ms of load-generator delay. Keep both values. Fail run when schedule lag exceeds preregistered client budget, because benchmark client, not server, became bottleneck.

One trap remains: a benchmark can reproduce arrivals perfectly while measuring startup noise instead of operating state.

Separate warmup, steady state, and drain

Warmup prepares mechanisms that production path expects to be warm. It may include model loading, tokenizer initialization, memory allocation, just-in-time compilation, CUDA Graph capture, connection setup, and scheduler stabilization. It must not secretly prefill prefix cache when production requests normally miss it.

Use phase boundaries that can be audited:

Don't start clock after queue is already full. Don't stop at final arrival and discard slow completions. Drained completions belong to request outcomes, but drain time must not turn an unstable arrival window into sustainable throughput. Track queue depth or unfinished-work slope during steady state.

A practical stability gate might require all of following, with values chosen before run:

• client schedule lag stays below budget;
• achieved arrival rate stays close to offered rate;
• queue depth has no persistent positive slope;
• completed rate doesn't keep falling across windows;
• error, timeout, and rejection rates remain within bounds;
• temperature, clocks, replica count, and worker restarts are recorded.

Cold-start and warm-capacity tests answer different questions. Publish separate curves rather than averaging them.

Baseline runs first from cold process. Candidate runs second with compiled kernels and populated prefix cache, then posts lower TTFT. What comparison should release review accept?

Answer

Neither raw result is attributable to candidate code. Declare cache and compilation state, apply same warmup policy to both systems, alternate run order, and repeat boundary points. Keep cold-start curve separate when startup behavior belongs to product contract.

Measure moments users can distinguish

Let request arrive at $a$, first output token appear at $t_1$, final token at $t_n$, and output contain $n$ tokens.

Time to first token (TTFT) includes queueing and prefill from chosen client boundary:

$$TTFT = t_1 - a$$

Inter-token latency (ITL), also called time between tokens, is each gap after first:

$$ITL_i = t_i - t_{i-1}, \qquad i=2,\ldots,n$$

Time per output token (TPOT) averages decode cadence after first token:

$$TPOT = \frac{t_n-t_1}{n-1}$$

For $n=1$, TPOT and ITL are undefined. Report those requests separately instead of dividing by zero or assigning zero.

End-to-end latency (E2E) covers full wait:

$$E2E = t_n-a$$

Consider one five-token response with arrival-relative token times [240, 280, 320, 440, 480] ms. TTFT is 240 ms. ITLs are [40, 40, 120, 40] ms. TPOT is $(480-240)/4=60$ ms, and E2E latency is 480 ms.

A 60 ms mean hides one 120 ms pause. That pause may or may not be visible because earlier tokens arrived ahead of playback pace. We need both distribution and sequence.

Percentiles need named populations

p99 ITL = 120 ms is incomplete. Did calculation pool every token gap, compute p99 per request then take p99 across requests, or use one request's gaps? Long responses dominate pooled-token distribution because they contribute more samples.

Choose denominator that matches promise:

Report sample count with every high percentile. A p99 from 40 requests is effectively one extreme observation, not stable tail estimate. Repeat runs or bootstrap intervals can expose that uncertainty, but they can't repair unrepresentative trace.

Fluidity respects token deadlines

Etalon proposes fluidity index, a deadline-based view of streaming. It sets a first-token deadline based on prefill size and a desired token cadence. Tokens produced early create slack. A late token consumes slack; if it misses deadline, later deadlines reset so one stall isn't counted forever.^{[6]Reference 6Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems.https://arxiv.org/abs/2407.07000}

For target TTFT 300 ms and token cadence 80 ms, nominal deadlines are 300, 380, 460, 540, and 620 ms. Our token times 240, 280, 320, 440, and 480 ms stay ahead of them. The raw 120 ms ITL exceeds 80 ms, but user already has buffered tokens, so this trace remains fluid under deadline model.

Now shift token times to 300, 380, 460, 580, and 620 ms. Average TPOT is still 80 ms, yet fourth token misses 540 ms deadline. Mean TPOT alone can't distinguish these experiences.

Fluidity doesn't replace TTFT, E2E, or raw ITL. It answers different question: did token stream meet playback deadlines after accounting for useful early work?

Metrics now describe one run. A release decision still needs curve over increasing load.

Find saturation on operating curve

Run same fixture at increasing offered rates. Preserve request order, shape mix, arrival-burst pattern, seed, warmup rule, and duration. Time-scaling realized timestamps is acceptable when goal is rate sweep and transformation is recorded.

The table below is a synthetic worked fixture, not hardware result. Its release contract requires at least 95% of requests to pass correctness, TTFT $\leq 300$ ms, and request-mean TPOT $\leq 80$ ms. Client schedule lag must remain below 20 ms and backlog slope at or below 0.02 requests per second. The p99 ITL column pools all token gaps; each response has four gaps, so every request has equal weight in this fixture.

Raw throughput keeps rising from 1.48 to 1.82 completed requests per second. Release capacity doesn't. At 2.0 offered RPS, queue grows and tail latency fails. Highest passing offered rate is 1.5 RPS.

DistServe defines goodput as maximum request rate served while meeting declared TTFT and TPOT SLO attainment goal per provisioned GPU.^{[7]Reference 7DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670} For broader release contract here:

$$\text{goodput}=\max_{\lambda}\left\{\lambda:\Pr[\text{success}\land\text{quality}\land\text{TTFT}\leq D_f\land\text{TPOT}\leq D_d]\geq\alpha\right\}$$

$\lambda$ is offered rate, $D_f$ is TTFT deadline, $D_d$ is decode deadline, and $\alpha$ is required attainment. Add ITL or fluidity gate when streaming smoothness belongs to product contract. Add stability predicates so short overloaded run can't pass by draining backlog later.

This equation reports fleet request rate. At stable point, offered and completed rates converge apart from bounded errors and rejections. Divide by fixed provisioned GPU count only when per-GPU goodput is comparison target.

At 2.0 offered RPS, system produces 1.46 contract-compliant completions per second, slightly above 1.41 at 1.5 RPS. Should release boundary move to 2.0 RPS?

Answer

No. Aggregate count of acceptable completions isn't same as request-level attainment. At 2.0 RPS, only 80% of requests pass 95% contract and backlog grows. Highest stable passing offered rate remains 1.5 RPS, with production operating point chosen below measured cliff.

Run one point below and one above selected boundary. Repeat boundary point, alternate baseline and candidate order, and sweep downward once when cache state or thermal history may cause hysteresis.

Compute percentiles and goodput from request records

The exercise below creates four deterministic synthetic load points. Each request carries scheduled arrival, token timestamps, and correctness. Nearest-rank percentile is explicit, as are per-request SLO and backlog gates.

At 2.0 RPS, three requests miss TTFT and one otherwise fast request fails quality, leaving 16 of 20 accepted. The predicate counts each request once even when one request violates several conditions.

Why does p99 ITL select single 71 ms maximum at 0.5 RPS? Each load point has 20 requests with four token gaps, so nearest-rank p99 of 80 pooled gaps selects 80th ordered value. Likewise, request-level p95 selects 19th of 20 requests. Larger samples are needed for stable tail claims. Code states population and convention so another implementation can reproduce them.

Change ATTAINMENT_TARGET to 0.99. The 1.5 RPS point fails because 19 of 20 is 95%, and fixture goodput drops to 1.0 RPS. SLO isn't decoration on graph. It changes answer.

Keep quality inside acceptance predicate

Serving candidate can alter output even when model weights stay fixed. Quantization changes numerical behavior. Scheduler order can expose nondeterminism. Speculative decoding, tokenizer changes, stop handling, truncation, or a malformed streaming parser can change result contract.

Use two linked suites:

1. Performance trace: representative arrival and shape fixture, with every response checked for protocol success, finish reason, token count, and schema validity.
2. Quality set: stable prompts with task-specific scoring, such as exact tool arguments, source-grounded claims, refusal behavior, structured-output validity, or human-reviewed rubric.

MLPerf treats quality as model's ability to produce correct outputs and requires valid benchmark run to meet scenario performance and quality constraints.^{[3]Reference 3MLPerf Inference Benchmark Rules.https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc} Carry same principle into custom benchmark without borrowing compliance label.

Don't count these as good requests:

• HTTP success with empty body;
• truncated stream missing terminal event;
• timeout completed during drain;
• gateway rejection or client cancellation;
• schema-valid answer with wrong tool argument;
• correct text produced after latency deadline when deadline is part of product contract.

For stochastic generation, exact string equality may be wrong gate. Pin decoding when possible, then score invariant behavior. Compare quality with enough examples to detect agreed regression. Performance curve is meaningless if candidate silently changed task.

Token telemetry must match protocol

Server-sent event chunk isn't necessarily one token. A server may buffer several tokens into one chunk, while tokenizer can split chunk text differently after Unicode boundaries. Chunk-to-chunk delay is valid client experience metric, but it isn't token ITL unless protocol guarantees token granularity or server emits token timestamps.

Name metric honestly:

Quality gate closes semantics. Operating curve closes capacity. Architecture must preserve both evidence paths.

Build benchmark as evidence pipeline

Fixture owns scheduled arrivals. Load generator records schedule lag and client timestamps. Serving system contributes queue, cache, scheduler, GPU, and worker events. Scorer joins them by immutable request ID, computes metrics, and emits a receipt on pass or a rejection with raw artifacts on failure.

Keep controller outside system under test. Pin network path or measure it explicitly. If baseline runs through local loopback while candidate crosses gateway and transport security, comparison changed scope.

Framework boundaries

No single tool owns whole contract:

Use tool for layer it controls. A polished CLI output isn't release receipt by itself.

Diagnose benchmark failures before tuning server

Retune one cause at time. A larger batch, new quantization, different replica count, and new load trace in same comparison yield no attribution.

Archive release receipt

Receipt should let another engineer answer three questions without contacting author: what ran, whether outputs remained acceptable, and where operating boundary sits.

Values in worked fixture aren't defaults for production. Set thresholds from product promise and baseline evidence. Select operating point below measured cliff with capacity margin for drift, failures, and traffic mix changes. Receipt should state margin rather than presenting boundary as safe production target.

Release protocol

One repeatable protocol is enough:

1. Pin baseline, candidate, hardware, model, tokenizer, decoding, and gateway scope.
2. Freeze request rows and realized open-loop timestamps; hash fixture.
3. Predeclare TTFT, TPOT or ITL/fluidity, quality, error, schedule-lag, and stability gates.
4. Warm only mechanisms expected warm in selected production scenario.
5. Replay identical trace at ascending load points, including one passing neighbor and one failing neighbor.
6. Alternate candidate order and repeat boundary points.
7. Join request events with queue, KV-cache, GPU, worker, and error telemetry.
8. Compute named percentiles, SLO attainment, and highest stable passing offered rate.
9. Run quality set and reject semantic regression before comparing performance.
10. Archive raw events, environment, curve, decision, operating margin, and rollback trigger.

A benchmark becomes engineering when it can refuse misleading win. Stable request trace supplies same question to each candidate. Operating curve exposes queueing cliff. Receipt preserves enough evidence to repeat decision after code, hardware, or traffic changes.