A/B Testing for LLMs

GPU Serving & Autoscaling gave you a fleet that can survive live traffic. A/B testing decides whether a large language model (LLM), prompt, routing, or retrieval change should receive that traffic at all.

A/B testing (also called split testing) is a randomized experiment that divides users into two groups: a control group that receives the existing version of a feature, and a treatment group that gets the new version^{[1]Reference 1Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}. By comparing outcomes between groups, you can determine whether a change improves the feature or if observed differences are just noise.

A developer-doc assistant for an internal platform currently uses a simple system prompt: "Be helpful." Your team proposes a new prompt: "Be helpful, cite the relevant API docs or runbook, and use the repository and service context from the request." It sounds better, but how do you prove it?

Three engineers preferring one reply proves nothing. You need a real experiment. A clicked citation, accepted runbook, or question resolved without human escalation is directly observable; LLM answer quality often needs a rubric or reviewer, and generated wording can vary across runs. A 1% quality gain isn't worth an unacceptable latency or safety regression.

You'll take a developer-doc assistant through a real online experiment, from hypothesis to verdict, while learning metric selection, sample sizing, traffic routing, and biases that can fool you into shipping a regression.

Why LLM A/B testing is different

Start with what LLM generation adds to ordinary online experimentation.

The scored-ticket analogy

Traditional A/B testing is like counting whether more users clicked a button. The event is objective. LLM A/B testing is like scoring source-grounded answers. You need a judge model or trained reviewer and a scorecard (a rubric) to turn subjective impressions into reproducible numbers.

Without that scorecard, you're just arguing about taste. With it, you can count how many times the judge prefers one response over another and whether the margin is large enough to matter.

Non-determinism

A/B tests already handle variable user outcomes: two developers assigned the same docs-search layout don't behave identically. LLM features add another variation source because the same prompt can yield different text as sampling, model serving, and context change. For offline comparisons, hold sampling and retrieval settings fixed and use deterministic decoding when the runtime supports it. If your stack exposes a seed, log it, but don't assume a seed alone makes live traffic reproducible. Otherwise, sampler or serving variation can obscure the treatment effect.

The subjectivity problem

Unlike a button click (binary), LLM quality is subjective. Two engineers might disagree on whether a response is "helpful." Your experiments must move from "it looks good" to rigorous, quantifiable metrics. That's why modern LLM evaluation uses a layered approach rather than gut feel.

The cascade effect

Small changes in a system prompt can lead to outsized regressions on edge cases that aren't immediately visible. A prompt tweak that improves the median query might catastrophically fail your hardest 5% of cases. This makes guardrail metrics and golden datasets essential, not optional.

What exactly are you testing?

Don't limit your experiments to "Model A vs Model B." LLM A/B tests can evaluate many variables:

In our docs-assistant example, the most common test is a prompt duel: the same model with two different system prompts. Understanding what you're testing determines your metrics, sample size, and success criteria.

Three levels of LLM evaluation

Before you pick a metric, you need to know what family of evaluation fits your task. These three levels move from cheap and deterministic to flexible and expensive.

Level 1: Rule-based checks

For structured outputs or code generation, deterministic rules are the fastest and most reliable evaluators.

• Exact match and regex: Does the output contain a valid JSON object with the expected keys?
• Code execution: Does the generated Python script run and pass a set of unit tests?
• Format compliance: Does the response follow the requested template?

These are cheap, fast, and objective. Use them whenever you can. They are also reference-based: you compare the output against a known correct answer or schema.

Level 2: Semantic similarity

When the right answer can be phrased many ways, word-overlap metrics like BLEU and ROUGE often miss the point. They reward verbatim copying, which is terrible for creative or conversational tasks.

A better approach is embedding-based similarity.

• BERTScore^{[2]Reference 2BERTScore: Evaluating Text Generation with BERT.https://arxiv.org/abs/1904.09675}: Uses contextual embeddings to compare token-level similarity, capturing paraphrases.
• Cosine similarity: Measures how close two sentence embeddings are in vector space.

These estimate whether the model's answer has similar meaning to a reference answer, even if the wording differs. They don't prove factual correctness or policy compliance. Like Level 1, they are reference-based: you still need a golden answer to compare against.

Level 3: LLM-as-judge

For open-ended conversation, even embeddings fall short because there's no single correct phrasing. A common evaluation pattern is to use a separately calibrated judge model to grade candidate responses against a rubric and any required source context.

• G-Eval^{[3]Reference 3G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.https://arxiv.org/abs/2303.16634}: A judge LLM scores outputs on a rubric (for example, Clarity 1-5, Accuracy 1-5).
• Pairwise comparison: The judge sees two responses and picks the better one based on explicit criteria.

This can avoid a single golden wording, but factual or policy tasks still need source context in the rubric. It's scalable and flexible, but judges have their own biases. Later sections return to those judge risks.

Reference-based methods (Levels 1 and 2) work best when you know what the output should look like. Reference-free methods (Level 3) are useful for creative, conversational, or open-ended tasks where the "right" answer isn't fixed.

Worked example: the system prompt duel

A complete offline experiment makes the evaluation contract concrete. Compare Prompt A ("Be helpful.") against Prompt B ("Be helpful, cite the relevant API doc or runbook, and use the repository context.") on a small golden dataset.

Step 1: Build a golden dataset

A golden dataset is a curated benchmark of your hardest product cases. Unlike generic benchmarks (like MMLU or HumanEval), a golden dataset reflects your specific use case: your developers' most complex questions and your platform's highest-risk workflows.

For the docs assistant, start with five representative questions:

These are edge cases that matter. Passing them screens important failures, but it doesn't prove behavior on routine traffic or unseen edge cases. Before an online A/B test, require the variant to pass an offline eval gate appropriate to its risk. This prevents obvious failures from consuming user traffic.

Step 2: Run the two prompts

Send each question through both system prompts. Record the outputs. For this offline comparison, request deterministic decoding where the selected runtime supports it and record all generation, retrieval, and model-version settings. Temperature zero reduces sampling variation in many runtimes; it doesn't by itself guarantee identical output across provider or backend changes.

In practice, you'd use an API client or a local model. Keep every parameter identical except the prompt itself, so you're measuring the prompt's effect, not sampler noise.

Step 3: Write the judge rubric

A rubric turns subjective quality into countable scores. Here's a simple three-axis rubric for our docs assistant:

You feed the judge LLM the question, the response, and this rubric, then ask it to score each axis from 1 to 5. The rubric turns the evaluation into a reviewable procedure: another engineer can run the same judge configuration and compare results, then measure agreement against held-out human labels.

Step 4: Score and calculate a winner

Suppose you run the five questions and get these average scores:

Prompt B wins on every axis. But notice that Prompt A never cited the source document, while Prompt B did so consistently. The rubric made that gap visible. Without the rubric, you might have glanced at the outputs and thought both were "fine."

In pairwise mode, you can also compute a win rate. If the judge compares A and B head-to-head on each question and declares B the winner 4 times, A the winner 0 times, and a tie 1 time, the win rate is:

$\text{Win Rate}_B = \frac{\text{Wins}_B + 0.5 \times \text{Ties}}{\text{Total Trials}} = \frac{4 + 0.5(1)}{5} = 0.90$

A 90% win rate over five examples is an encouraging screening signal on this curated set, not strong launch evidence. You would still need broader offline checks and a live A/B test to determine user impact, while this small duel can filter out obvious losers before you risk real traffic.

A golden dataset plus a clear rubric turns "looks good" into recorded screening evidence. Use an offline gate before live exposure whenever the treatment can change answer quality or safety.

Metrics that matter in production

Offline rubrics are useful, but production experiments need metrics that reflect real value and system health. Successful tests use a taxonomy of metrics organized into three tiers based on how they're measured.

Keep latency terms straight, and document the formulas your tooling uses. TTFT is the delay until the first token arrives. ITL usually measures time between consecutive output tokens and is often called TPOT. Some systems also expose a separate request-level average with a different denominator. Mixing those up makes latency regressions hard to diagnose because each metric points to a different bottleneck.

Within each experiment, further categorize by priority:

1. Primary metrics: Direct business value. For a docs assistant, this could be "question resolved without human escalation" or developer satisfaction score.
2. Guardrail metrics: Predefined constraints. For example, a sustained p95 latency breach or statistically credible safety degradation can block promotion regardless of quality gains; exact policy thresholds depend on your baseline and SLO.
3. Secondary metrics: Debugging signals that explain why the primary metric moved, such as whether accepted code was shorter, more complete, or easier to review.

For LLM documents, cost belongs in the same dashboard as quality and latency. A variant can "win" by generating longer answers, invoking more tools, or consuming more retrieved context. Track cost per response, cost per successful session, and output-token growth as guardrails, not as an after-the-fact finance metric.

This Python dictionary categorizes typical metrics for evaluating our docs assistant, then applies a simple promotion policy to observed deltas. A positive primary result isn't enough when a predefined guardrail fails.

How many threads do you need?

Detecting a small improvement in LLM quality is like hearing a whisper in a noisy room. You need to listen longer (collect more data) to be sure it wasn't just random noise. If your metric has high variance (like token usage) or low signal (like thumbs up/down), you need a larger sample size to distinguish signal from noise.

Calculate the required sample size before starting so the test has statistical power (the probability of detecting a real effect).

A numeric warm-up

Suppose your docs assistant currently resolves 60% of threads on its own (the baseline rate). A five-point improvement to 65% is the smallest lift worth launching. How many threads do you need in each group to design a test with useful power for that target?

With a standard 5% false-positive rate and 80% power, the normal approximation gives roughly 1,471 threads per arm, or about 2,942 total. That's a lot for a low-traffic internal tool, which is why offline screening with a golden dataset matters. It lets you kill bad variants before they consume live traffic.

Notice what happens if you aim smaller. Detecting a 2-point lift (from 60% to 62%) balloons the requirement to over 18,600 total threads. Smaller effects require quadratically more samples.

The formula

For binary metrics (like "resolved" vs "not resolved"), a common two-sided normal approximation for a 50/50 test is:

$n_{\text{per arm}} \approx \frac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{\beta}\sqrt{p_A(1-p_A) + p_B(1-p_B)}\right)^2}{(p_B - p_A)^2}$

Here, $p_A$ is the baseline rate, $p_B = p_A + \delta$ is the treatment rate implied by your minimum detectable effect (MDE), and $\bar{p} = (p_A + p_B)/2$. The $z$ values come from the standard normal distribution: $z_{\alpha/2}$ controls false positives and $z_\beta$ controls false negatives.

Python calculator

This function computes the necessary sample size for a binary metric. It takes the baseline conversion rate, the minimum detectable effect, alpha, and power as inputs, and returns the total required sample size using the standard normal approximation.

Use this calculator before launch, not after results arrive. For the docs-assistant example, a 5-point lift needs thousands of threads, while a 2-point lift needs far more.

For judge scores, latency, or token counts, estimate variance from historical logs or a pilot run and use the matching power analysis for continuous metrics. Don't reuse a binary-proportion shortcut for everything.

Routing users to the right variant

The randomization unit determines how traffic and users are divided between the control and treatment groups. It affects both the statistical validity of the experiment and the consistency of the user experience.

For chat-like assistants, randomizing per user or per conversation is usually the right default. Pin that assignment for the full thread so follow-up turns, tool calls, and regenerations all hit the same variant. This keeps context and tone consistent, and it reduces within-session contamination where one bad turn changes how the user behaves on later turns.

The no-interference part of SUTVA (Stable Unit Treatment Value Assumption) can fail in collaborative features or shared documents because one user's assignment can affect another user's outcome. SUTVA also requires a well-defined treatment: don't hide materially different variants behind the same experiment label. When users influence each other, you often need cluster-based randomization rather than simple per-user hashing.

Session pinning

Stable assignment sounds simple until you ship a multi-turn assistant. In practice, you need a deterministic routing key (for example user_id or conversation_id), persistent storage of the assigned variant, and telemetry that logs that assignment on every turn. If a conversation starts on Variant B and the second turn accidentally lands on Variant A, you haven't just damaged UX. You've invalidated the experiment because prompt state, retrieved context, and prior model behavior now leak across variants.

This deterministic router assigns at the conversation level. Each turn of one docs-assistant thread gets the same prompt variant, and the logged assignment can travel with every trace:

Shadow, canary, and live A/B tests

Different rollout patterns answer different questions:

For high-risk changes, a common escalation sequence is offline evaluation, then shadow, then canary, then a user-facing A/B test. A low-risk copy adjustment may not need every step; record the reason for skipping one.

Cache locality and cold-start bias

LLM serving is stateful in ways normal web experiments aren't. Prefix caches, KV cache pages, and warm GPU workers strongly influence TTFT and throughput^{[5]Reference 5Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}. A shadow deployment doubles inference work, and a low-traffic canary can look artificially slow because it misses warm-cache reuse or lands on cold replicas. Measure warm and cold latency separately, and make sure the treatment has enough steady traffic to stay warm before you call a latency regression "real."

Interleaving: a blind taste test for ranking

For ranking developer-doc results, standard A/B tests compare outcomes across separate traffic groups. Interleaving is a within-query design^{[6]Reference 6Large-scale Validation and Analysis of Interleaved Search Evaluation.https://doi.org/10.1145/2094072.2094078}: it mixes candidates from both rankers into one list and attributes clicks back to each source. In the large-scale search experiments reported by Chapelle et al., interleaving was substantially more sensitive than A/B comparison for detecting ranking differences. That advantage is task-specific: it applies when both systems can safely contribute items to one ranking, not to two different free-form chatbot responses.

In practice, the system mixes results from both models and measures user preference by which results get clicked. A concrete example for developer-doc search:

Because interleaving compares rankers on the same query for the same user, it controls much of the query and user-intent variation within that trial. That can make it more sensitive than a standard A/B test for ranking problems.

Team draft interleaving

The "Team Draft" method builds a balanced mix of results. Imagine two team captains (Model A and Model B) picking their best remaining unseen result. When both teams have contributed the same number of items, choose the next captain randomly. Otherwise, the team with fewer items picks next.

This implementation follows that Team Draft rule. It accepts two ranked lists of results as inputs and outputs a single interleaved list along with the items attributed to each model. If one ranker has no unseen item left, the other ranker keeps contributing until it also exhausts or the list reaches k; one short or duplicate-heavy list must not truncate valid results from the other.

Statistical rigor

Multiple testing correction

If you searched through 20 independent secondary metrics at $\alpha=0.05$, you would have about a 64% chance of finding at least one "significant" result purely by chance (Family-Wise Error Rate, or FWER). Real experiment metrics are often correlated, so the exact number changes, but the core problem doesn't. Predeclare the primary decision metric; when you interpret a family of exploratory metrics, segments, or rubric axes, apply an appropriate correction such as Benjamini-Hochberg^{[7]Reference 7Controlling the false discovery rate: a practical and powerful approach to multiple testing.https://doi.org/10.1111/j.2517-6161.1995.tb02031.x} to control the False Discovery Rate (FDR).

This example applies multiple testing corrections to experimental results. It takes a dictionary of metric names and their corresponding test statistics and p-values, and returns the adjusted significance outcomes.

Here resolution_rate survives Benjamini-Hochberg but not Bonferroni because another related metric is even stronger. latency_p95 and cost_per_session fail both corrections.

Sequential testing

Fixed-horizon A/B tests require choosing the sample size $N$ in advance and avoiding early stopping based on ordinary p-values. Checking results daily and stopping whenever they cross $p < 0.05$ inflates the false-positive rate. A registered sequential design^{[8]Reference 8Peeking at A/B Tests: Why it matters, and what to do about it.https://doi.org/10.1145/3097983.3097992} permits planned interim looks by allocating error budget across those looks, allowing an early decision only when its boundary is crossed.

One approach uses a spending function such as O'Brien-Fleming to allocate total error budget across planned checkpoints. Early checks require stronger evidence. Exact critical values depend on the design, sidedness, information fraction, and analysis method; generate them with a statistical package and register them before traffic starts.

Repeatedly checking p-values without a sequential testing correction inflates your false positive rate. The exact inflation depends on the metric, test, and correlation between looks, which is why sequential methods matter.

The monitoring code below deliberately consumes precomputed boundaries instead of pretending to calculate a valid sequential test. It records four registered looks and stops only when the observed statistic exceeds that look's critical value:

Guardrail monitoring

Guardrails protect users and your serving environment during exposure. Different signals require different actions. An inline personally identifiable information (PII) or unsafe-content detector can block one response immediately. Sampled judge safety rate, p95 latency, or average cost is an aggregate estimate with noise; it usually needs a defined window and persistence rule before you pause or roll back a treatment. If a required guardrail is missing or non-finite, the window is inconclusive and exposure pauses; the missing value isn't a threshold pass.

Setting appropriate thresholds requires balancing safety with experimental velocity. If thresholds are too tight, normal statistical noise will trigger false alarms and halt valid experiments. If they're too loose, users might be exposed to harmful model degradation before the system reacts.

Set guardrails from the baseline distribution you observed during stable periods, not from arbitrary fixed values. This reduces false rollbacks caused by normal day-to-day traffic swings.

This Python class monitors aggregate windows. A single breached window raises an alert; only a sustained breach reaches the predeclared action. In a real system, keep synchronous per-response blocking outside this aggregate loop.

One complete window alerts without making an aggregate decision. A second sustained window reaches each registered action; an objective sustained error breach triggers rollback while latency and cost cause pause or review. In the final window, required cost telemetry is absent, so the result is inconclusive and exposure pauses immediately.

Multi-armed bandits

Standard A/B tests explore first (to find a winner) and exploit later (when you ship the winner to 100% of users). But what if the exploration phase is too costly? If a bad model variant causes users to churn, you want to stop sending traffic to it as quickly as possible.

Multi-armed bandits (MAB) dynamically adjust traffic allocation during the experiment based on real-time performance. Using algorithms such as Thompson Sampling, the system routes more traffic to variants that appear to be winning and less traffic to variants that appear weak. This minimizes "regret," which is the total penalty incurred by exposing users to suboptimal models. Bandits fit continuous optimization tasks (such as prompt tuning) where you have many variants and aim to automatically deprecate underperforming ones without waiting for a fixed-horizon A/B test.

But MABs introduce architectural complexity. They require a tightly coupled feedback loop where user actions (such as a thumbs up) immediately update the routing logic. In distributed systems with high latency or delayed rewards (for example, measuring 7-day retention), bandits can be challenging to implement correctly compared to static A/B test splits.

This Python example implements a basic Beta-Bernoulli Thompson Sampling bandit. It assumes a binary reward signal such as click/no-click or accept/reject. If your reward is continuous or heavily delayed, the posterior update changes.

Biases that distort results

Generative outputs and filtered trace analysis make several familiar experiment biases easy to miss. None is exclusive to LLMs, but each can reverse an apparent winner if you ignore it.

Position bias

Human reviewers and judge models can favor the first option shown. Published LLM-judge evaluations report position bias^{[9]Reference 9Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}, and pairwise human review has the same measurement risk when presentation order isn't balanced.

Solution

Always randomize the order of presentation. If you're showing two model outputs A and B to reviewers, put A first for 50% of comparisons and B first for the other 50%. Log the display order to control for this bias during analysis.

Running a head-to-head judge comparison with A always first can make order effects look like model quality. Swap positions and analyze order effects so presentation bias is measured and reduced rather than silently attributed to the model.

Novelty effect

Users initially engage more with new models or features just because they're different, not necessarily better. A new citation-panel feature might see a spike in usage on Day 1 that evaporates by Day 7.

Solution

Plan duration around known demand cycles and novelty risk. A full weekly cycle is a reasonable starting point when weekday and weekend traffic differ, but it isn't a universal stopping rule. If you want a burn-in window, define it before launch and exclude it consistently. Don't decide to drop early days after seeing a spike.

Simpson's paradox

A model can look better overall even while losing inside every major segment. This usually happens when the analyzed sample has different segment mixes across variants.

Both segment-level comparisons favor Model A, but the aggregate flips because Model B's analyzed sample contains far more mobile users (who have higher resolution rates overall). That's Simpson's paradox.

How to avoid it

Always segment results by pre-treatment user groups (for example, tenure, subscription tier, device type) and inspect the sample mix in each arm. Be especially careful with triggered analyses, missing-label subsets, or any filter that treatment can influence. Before declaring a winner, verify that the treatment doesn't severely degrade the experience for critical cohorts, even if the aggregate top-line metric improves.

The calculation below reproduces the aggregate reversal. Both pre-treatment cohorts favor Model A even though Model B's mobile-heavy analyzed sample wins after pooling:

LLM-as-judge for offline screening

LLM-as-judge^{[9]Reference 9Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} offers automated quality comparison to screen candidates before committing to a live A/B test. Full A/B tests with human evaluation are expensive and slow.

LLM judges are useful triage systems, not ground truth. They can cheaply filter obvious regressions in offline evaluation or shadow mode, but they also inherit position bias, verbosity bias, and judge-model bias^{[9]Reference 9Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}. Use them to narrow the field before you spend live traffic.

The asynchronous function below uses a strong LLM to evaluate two competing model outputs. It takes the original prompt and both responses as inputs, and returns a judgment ("A", "B", or "TIE") with a brief justification. Candidate outputs are untrusted data. A judge harness should isolate them from evaluator instructions with structured messages or a structured-input API, then test prompt-injection cases before using the judge for decisions.

Key practices for LLM-as-judge

• Swap positions: To measure position bias, run comparisons in both A/B and B/A order.
• Use a capable judge candidate: Greater capability may help, but calibration against human labels decides whether the judge is usable.
• Calibrate against humans: Measure judge agreement on a held-out human-labeled set before you trust it for go/no-go decisions.
• Audit length bias: Slice wins by response length and refusal style to catch judges that reward verbosity instead of correctness.
• Don't replace A/B tests entirely: Use LLM-as-Judge to filter candidates, then validate the winning candidate with a real A/B test on live traffic.

Published judge evaluations report position and verbosity biases.^{[9]Reference 9Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} Check whether your judge's wins correlate with display order or response length before trusting its verdict.

Online evaluation and the offline-online gap

Everything above the live A/B test (golden datasets, rubrics, offline judges) measures what your variant does on a frozen set you chose. Production measures what it does on traffic you didn't choose. That difference is the offline-online gap, and closing that gap is the whole point of online evaluation. A variant can win 90% offline and still regress live because the real query mix is different, latency and cost shift under load, and rare safety failures only surface in production.

A common production pattern is to keep the judge running after launch instead of only before it. You sample a predeclared slice of live traffic sized for budget and detection needs, attach the request, retrieved context, tool calls, and output to a trace, then score that trace asynchronously with the same rubric you used offline. Because the eval is linked to a trace, a quality drop points you straight to the failing request rather than to an aggregate dashboard with no drill-down. This is the trace-linked online judge.

Keep two ideas separate, because production systems implement them differently:

An inline guardrail is a safety gate that must run before the response ships. An online judge is a measurement system that runs after, on a sample; when it's kept off the request path, it doesn't add user-facing latency. Confusing the two leads people to either slow every request with a judge call or to treat a slow async score as a real-time block.

Online judges also drift when the judge version, rubric, prompt, or traffic mix changes. Use the same calibration discipline you use offline, run continuously. Track judge-vs-human agreement on a rotating audit set, choose an acceptance threshold for your decision risk, log the full judge configuration on every score, and re-calibrate whenever that configuration changes.

This audit check uses an explicit local policy rather than a universal agreement cutoff:

Tooling

Managing LLM experiments at scale usually requires more than spreadsheets and ad hoc notebooks. Common choices include:

When selecting tools, prioritize integration with your existing ML infrastructure, immutable experiment definitions, trace linkage, and guardrail alerting. A useful toolchain connects offline evaluation to online experiments with shared metric definitions.

Practice: build a prompt arena

Your turn. Build a small "Prompt Arena" CLI tool that automates the workflow we walked through.

Requirements

1. Create a CSV file with 10 docs-assistant questions (use the five golden questions above plus five more of your own).
2. Write two system prompts: a baseline and a treatment.
3. Send each question through both prompts using an API client or local model runtime.
4. Use a judge candidate or human reviewers to score each response on Clarity, Accuracy, and Source Citation, then record how that evaluator was calibrated.
5. Print a summary report: average scores per prompt, win rate, and which questions had the biggest gaps.

Check your understanding

• If Prompt A scores higher on Accuracy but lower on Source Citation, which metric is more important for a docs assistant? (Answer: You must define that policy before seeing results. Factual source correctness is a hard requirement; source citation completeness may be primary or a guardrail depending on docs policy.)
• You run 50 threads and the treatment wins 60% of the time. Is that enough to ship? (Answer: Probably not. 50 threads is far below the sample size needed for a 5-point lift at standard power. Run a power analysis.)
• Why is per-request randomization dangerous in a multi-turn docs assistant? (Answer: A user might see different tones, source context, or tool state on each turn, which breaks the experiment and creates a terrible experience.)

Mastery check

The Inference & Production Scale loop is complete: serve a model fleet, autoscale it, route and fall back across providers, watch cost and tokens, observe it in production, and decide what gets to ship through online evaluation. That last decision belongs to the online-evaluation plan; defend the full plan, not a p-value alone.

Evaluation rubric

1. Design a rubric for a specific business use case and explain the difference between reference-based and reference-free evaluation.
2. Identify why a per-request randomization breaks a multi-turn conversation experiment.
3. Calculate a required sample size for a binary metric and explain why small samples lead to false positives.
4. Name three deployment patterns (shadow, canary, live A/B) and describe when each is appropriate.
5. Spot position bias, novelty effects, and Simpson's paradox in experimental results.

Also check that the plan can:

• Pick a randomization unit that avoids SUTVA violations. For collaborative documents, that often means team, organization, document cluster, or switchback randomization instead of per-user randomization.
• Separate primary, secondary, and guardrail metrics. Resolution rate can be primary; safety rate, p95 latency, error rate, and cost per successful session are guardrails.
• Size the test before launch. A tiny golden dataset is useful screening evidence, not launch evidence.
• Choose shadow, canary, or live A/B testing based on risk and measurement goals.
• Add predefined circuit breakers for objective critical failures and persistence rules for noisy aggregate guardrails.
• Account for cache, cold-start, output-length, and routing bias before blaming the model.
• Choose interleaving for ranking comparisons where both systems can be shown in one mixed result list.
• Apply multiplicity corrections when many metrics, segments, or rubrics are tested at once.

Follow-up questions

How do you handle network effects in LLM A/B tests? Network effects break simple A/B tests when one user's treatment changes another user's experience. Shared AI-edited documents are a typical example: User A's improved edits can make User B happier even if User B is in the control group. Use cluster randomization at the team, organization, or document-cluster level. For short-lived effects, switchback experiments can also work.

Which common guardrails should an LLM feature consider? Typical choices include p95/p99 latency, error rate, empty-response rate, safety violations, personally identifiable information (PII) leakage, and cost per successful session. Block objective per-response safety failures inline; pause or roll back exposure according to predefined persistence rules for aggregate signals.

How do you test multi-turn conversational effects? Randomize by session or conversation, not by request. Track conversation-level outcomes such as resolution rate, human handoff, sentiment trajectory, and retention after early turns. A bad first answer can poison every later turn.

When would you use a bandit instead of an A/B test? Use bandits to optimize cumulative reward during exploration, such as continuous prompt tuning with fast feedback. Use classical A/B tests when you need clean inference, fixed exposure, long-term outcomes, or high-confidence launch evidence.

Common pitfalls

• Using vanity metrics instead of action metrics tied to user value.
• Randomizing every request in a multi-turn assistant.
• Skipping sample-size and minimum-detectable-effect planning.
• Ignoring cost per session and output-length growth.
• Treating every metric and segment as independent without multiplicity correction.
• Calling cold-start or cache-locality effects model regressions.
• Running without safety, reliability, and cost guardrails.
• Comparing judge outputs with A always first instead of swapping positions.
• Declaring victory from a tiny golden dataset without live validation.

If you can defend those decisions, you have closed out production-scale LLM operations. Next comes System Design Capstones, where you assemble serving, observability, guardrails, and online evaluation into one end-to-end design under interview conditions. The first capstone, a real-time content moderation system, leans directly on the guardrail and online-evaluation thinking above: it has to keep user-generated content safe at scale using LLMs and specialized classifiers.

Practice drill

Write an experiment launch plan for a docs-assistant treatment:

1. Define primary metric, guardrails, assignment unit, sample-size target, and stopping rule before traffic starts.
2. Add offline gate results, shadow-mode checks, and live canary thresholds.
3. Create a metric table with resolution rate, safety rate, p95 latency, cost per session, judge agreement, and false refusal rate.
4. Write the exact ship, hold, and rollback decisions for three possible outcomes, including one primary-metric win that breaches guardrails.

The artifact should be enough for a reviewer to tell whether the experiment is valid before seeing results.