A/B Testing for LLMs

GPU Serving & Autoscaling gave you a fleet that can survive production traffic. A/B testing decides whether a large language model (LLM), prompt, routing, or retrieval change should receive that traffic at all.

A/B testing (also called split testing) is a randomized experiment that divides users into two groups: a control group that receives the existing version of a product, and a treatment group that gets the new version^[1]. By comparing outcomes between groups, you can determine whether a change actually improves the product or if observed differences are just noise.

Imagine you run the customer support bot for an online retailer. Right now the bot uses a simple system prompt: "Be helpful." Your team proposes a new prompt: "Be helpful, cite the relevant return or shipping policy, and use the customer's name from the order context." The new prompt sounds better, but how do you prove it?

You can't just ask three engineers which replies they prefer. You need a rigorous experiment. A click or completed return is directly observable; LLM answer quality often needs a rubric or reviewer, and generated wording can vary across runs. A 1% improvement in a quality score isn't valuable if it comes with an unacceptable latency increase or a safety regression.

This article walks you through designing a real online experiment for an LLM-powered feature. You'll build a concrete test from hypothesis to verdict, using a customer support bot as the running example. Along the way you'll learn metric selection, statistical sizing, traffic routing, and the biases that can fool you into shipping a regression.

Why LLM A/B testing is different

Before diving into experimental design, you need to understand what LLM generation adds to ordinary online experimentation.

The scored-ticket analogy

Traditional A/B testing is like counting whether more users clicked a button. The event is objective. LLM A/B testing is like scoring support-ticket replies. You need a judge model or trained reviewer and a scorecard (a rubric) to turn subjective impressions into reproducible numbers.

Without that scorecard, you're just arguing about taste. With it, you can count how many times the judge prefers one response over another and whether the margin is large enough to matter.

Non-determinism

A/B tests already handle variable user outcomes: two users assigned the same checkout button don't behave identically. LLM features add another variation source because the same prompt can yield different text as sampling, model serving, and context change. For offline comparisons, hold sampling and retrieval settings fixed and use deterministic decoding when the runtime supports it. If your stack exposes a seed, log it, but don't assume a seed alone makes live traffic reproducible. Otherwise, sampler or serving variation can obscure the treatment effect.

The subjectivity problem

Unlike a button click (binary), LLM quality is subjective. Two engineers might disagree on whether a response is "helpful." Your experiments must move from "it looks good" to rigorous, quantifiable metrics. That's why modern LLM evaluation uses a layered approach rather than gut feel.

The cascade effect

Small changes in a system prompt can lead to outsized regressions on edge cases that aren't immediately visible. A prompt tweak that improves the median query might catastrophically fail your hardest 5% of cases. This makes guardrail metrics and golden datasets essential, not optional.

What exactly are you testing?

Don't limit your experiments to "Model A vs Model B." LLM A/B tests can evaluate many variables:

In our support bot example, the most common test is a prompt duel: the same model with two different system prompts. Understanding what you're testing determines your metrics, sample size, and success criteria.

Three levels of LLM evaluation

Before you pick a metric, you need to know what family of evaluation fits your task. Think of these as three levels of rigor, from cheap and deterministic to flexible and expensive.

Level 1: Rule-based checks

For structured outputs or code generation, deterministic rules are the fastest and most reliable evaluators.

• Exact match and regex: Does the output contain a valid JSON object with the expected keys?
• Code execution: Does the generated Python script run and pass a set of unit tests?
• Format compliance: Does the response follow the requested template?

These are cheap, fast, and objective. Use them whenever you can. They are also reference-based: you compare the output against a known correct answer or schema.

Level 2: Semantic similarity

When the right answer can be phrased many ways, word-overlap metrics like BLEU and ROUGE often miss the point. They reward verbatim copying, which is terrible for creative or conversational tasks.

A better approach is embedding-based similarity.

• BERTScore^[2]: Uses contextual embeddings to compare token-level similarity, capturing paraphrases.
• Cosine similarity: Measures how close two sentence embeddings are in vector space.

These tell you whether the model's answer means the same thing as a reference answer, even if the wording differs. Like Level 1, they are reference-based: you still need a golden answer to compare against.

Level 3: LLM-as-judge

For open-ended conversation, even embeddings fall short because there's no single correct phrasing. A common evaluation pattern is to use a separately calibrated judge model to grade candidate responses against a rubric and any required source context.

• G-Eval^[3]: A judge LLM scores outputs on a rubric (for example, Clarity 1-5, Accuracy 1-5).
• Pairwise comparison: The judge sees two responses and picks the better one based on explicit criteria.

This can avoid a single golden wording, but factual or policy tasks still need source context in the rubric. It's scalable and flexible, but judges have their own biases. We'll return to those later.

Reference-based methods (Levels 1 and 2) work best when you know what the output should look like. Reference-free methods (Level 3) are necessary for creative, conversational, or open-ended tasks where the "right" answer isn't fixed.

Worked example: the system prompt duel

Let's walk through a complete offline experiment for our support bot. The goal is to compare Prompt A ("Be helpful.") against Prompt B ("Be helpful, cite the relevant policy, and use the customer's name.") on a small golden dataset.

Step 1: Build a golden dataset

A golden dataset is a curated benchmark of your hardest real-world cases. Unlike generic benchmarks (like MMLU or HumanEval), a golden dataset reflects your specific use case: your customers' most complex queries, your product's most critical workflows.

For the support bot, start with five representative questions:

These are edge cases that matter. Passing them screens important failures, but it doesn't prove behavior on routine traffic or unseen edge cases. Before an online A/B test, require the variant to pass an offline gate appropriate to its risk. This prevents obvious failures from consuming user traffic.

Step 2: Run the two prompts

Send each question through both system prompts. Record the outputs. For this offline comparison, request deterministic decoding where the selected runtime supports it and record all generation, retrieval, and model-version settings. Temperature zero reduces sampling variation in many runtimes; it doesn't by itself guarantee identical output across provider or backend changes.

In practice, you'd use an API client or a local model. The key is to keep every parameter identical except the prompt itself, so you're measuring the prompt's effect, not sampler noise.

Step 3: Write the judge rubric

A rubric turns subjective quality into countable scores. Here's a simple three-axis rubric for our support bot:

You feed the judge LLM the question, the response, and this rubric, then ask it to score each axis from 1 to 5. The rubric is what makes the evaluation reproducible: another engineer can run the same judge with the same rubric and get comparable numbers.

Step 4: Score and calculate a winner

Suppose you run the five questions and get these average scores:

Prompt B wins on every axis. But notice that Prompt A never cited the policy, while Prompt B did so consistently. The rubric made that gap visible. Without the rubric, you might have glanced at the outputs and thought both were "fine."

In pairwise mode, you can also compute a win rate. If the judge compares A and B head-to-head on each question and declares B the winner 4 times, A the winner 0 times, and a tie 1 time, the win rate is:

$\text{Win Rate}_B = \frac{\text{Wins}_B + 0.5 \times \text{Ties}}{\text{Total Trials}} = \frac{4 + 0.5(1)}{5} = 0.90$

A 90% win rate over five examples is an encouraging screening signal on this curated set, not strong launch evidence. You would still need broader offline checks and a live A/B test to determine user impact, while this small duel can filter out obvious losers before you risk real traffic.

A golden dataset plus a clear rubric turns "looks good" into recorded screening evidence. Use an offline gate before live exposure whenever the treatment can change answer quality or safety.

Metrics that matter in production

Offline rubrics are useful, but production experiments need metrics that reflect real business value and system health. Successful tests use a taxonomy of metrics organized into three tiers based on how they're measured.

Keep latency terms straight. TTFT is the delay until the first token arrives. TPOT is the average time per generated token after streaming starts. ITL is the gap between consecutive output tokens. Mixing those up makes latency regressions hard to diagnose because each metric points to a different bottleneck.

Within each experiment, further categorize by priority:

1. Primary metrics: Direct business value. For a support bot, this could be "conversation resolved without agent handoff" or customer satisfaction score.
2. Guardrail metrics: Predefined constraints. For example, a sustained latency breach or statistically credible safety degradation can block promotion regardless of quality gains; exact policy thresholds depend on your baseline and SLO.
3. Secondary metrics: Debugging signals. Why did users accept the code? Was it shorter? More complex? These help you understand why the primary metric moved.

For LLM products, cost belongs in the same dashboard as quality and latency. A variant can "win" by generating longer answers, invoking more tools, or consuming more retrieved context. Track cost per response, cost per successful session, and output-token growth as guardrails, not as an after-the-fact finance metric.

The following Python dictionary categorizes typical metrics for evaluating our support bot, then applies a simple promotion policy to observed deltas. A positive primary result isn't enough when a predefined guardrail fails.

How many conversations do you need?

Detecting a small improvement in LLM quality is like hearing a whisper in a noisy room. You need to listen longer (collect more data) to be sure it wasn't just random noise. If your metric has high variance (like token usage) or low signal (like thumbs up/down), you need a larger sample size to distinguish signal from noise.

You need to calculate the required sample size before starting to ensure statistical power (the probability of detecting a real effect).

A numeric warm-up

Suppose your support bot currently resolves 60% of chats on its own (the baseline rate). A five-point improvement to 65% is the smallest lift worth launching. How many conversations do you need in each group to design a test with useful power for that target?

With a standard 5% false-positive rate and 80% power, the normal approximation gives roughly 1,471 conversations per arm, or about 2,942 total. That's a lot for a small support team, which is why offline screening with a golden dataset is so valuable. It lets you kill bad variants before they consume live traffic.

Notice what happens if you aim smaller. Detecting a 2-point lift (from 60% to 62%) balloons the requirement to over 18,600 total conversations. Smaller effects require quadratically more samples.

The formula

For binary metrics (like "resolved" vs "not resolved"), a common two-sided normal approximation for a 50/50 test is:

$n_{\text{per arm}} \approx \frac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{\beta}\sqrt{p_A(1-p_A) + p_B(1-p_B)}\right)^2}{(p_B - p_A)^2}$

Here, $p_A$ is the baseline rate, $p_B = p_A + \delta$ is the treatment rate implied by your minimum detectable effect (MDE), and $\bar{p} = (p_A + p_B)/2$. The $z$ values come from the standard normal distribution: $z_{\alpha/2}$ controls false positives and $z_\beta$ controls false negatives.

Python calculator

The following function computes the necessary sample size for a binary metric. It takes the baseline conversion rate, the minimum detectable effect, alpha, and power as inputs, and returns the total required sample size using the standard normal approximation.

Use this calculator before launch, not after results arrive. For the support-bot example, a 5-point lift needs thousands of conversations, while a 2-point lift needs far more.

For judge scores, latency, or token counts, estimate variance from historical logs or a pilot run and use the matching power analysis for continuous metrics. Don't reuse a binary-proportion shortcut for everything.

Routing users to the right variant

The randomization unit determines how traffic and users are divided between the control and treatment groups. Choosing the right unit is critical for both the statistical validity of the experiment and the consistency of the user experience.

For chat products, randomizing per user or per conversation is usually the right default. Pin that assignment for the full thread so follow-up turns, tool calls, and regenerations all hit the same variant. This keeps context and tone consistent, and it reduces within-session contamination where one bad turn changes how the user behaves on later turns.

SUTVA (Stable Unit Treatment Value Assumption) can still fail in collaborative features or shared documents, because users can influence each other. In those cases, you often need cluster-based randomization rather than simple per-user hashing.

Session pinning

Stable assignment sounds simple until you ship a multi-turn product. In practice, you need a deterministic routing key (for example user_id or conversation_id), persistent storage of the assigned variant, and telemetry that logs that assignment on every turn. If a conversation starts on Variant B and the second turn accidentally lands on Variant A, you haven't just damaged UX. You've invalidated the experiment because prompt state, retrieved context, and prior model behavior now leak across variants.

This deterministic router assigns at the conversation level. Each turn of one return inquiry gets the same prompt variant, and the logged assignment can travel with every trace:

Shadow, canary, and live A/B tests

Different rollout patterns answer different questions:

For high-risk changes, a common escalation sequence is offline evaluation, then shadow, then canary, then a user-facing A/B test. A low-risk copy adjustment may not need every step; record the reason for skipping one.

Cache locality and cold-start bias

LLM serving is stateful in ways normal web experiments aren't. Prefix caches, KV cache pages, and warm GPU workers strongly influence TTFT and throughput^[5]. A shadow deployment doubles inference work, and a low-traffic canary can look artificially slow simply because it misses warm-cache reuse or lands on cold replicas. Measure warm and cold latency separately, and make sure the treatment has enough steady traffic to stay warm before you call a latency regression "real."

Interleaving: a blind taste test for ranking

For ranking products, standard A/B tests compare outcomes across separate traffic groups. Interleaving is a within-query design^[6]: it mixes candidates from both rankers into one list and attributes clicks back to each source. In the large-scale search experiments reported by Chapelle et al., interleaving was substantially more sensitive than A/B comparison for detecting ranking differences. That advantage is task-specific: it applies when both systems can safely contribute items to one ranking, not to two different free-form chatbot responses.

In practice, the system mixes results from both models and measures user preference by which results get clicked. A concrete example for an e-commerce search:

Because interleaving compares rankers on the same query for the same user, it controls much of the query and user-intent variation within that trial. That can make it more sensitive than a standard A/B test for ranking problems.

Team draft interleaving

The "Team Draft" method ensures a fair and balanced mix of results. Imagine two team captains (Model A and Model B) taking turns picking their best remaining result to place in the final list.

This implementation illustrates the Team Draft interleaving strategy. It accepts two ranked lists of results as inputs and outputs a single interleaved list along with the items attributed to each model, ensuring a balanced representation.

Statistical rigor

Multiple testing correction

If you searched through 20 independent secondary metrics at $\alpha=0.05$, you would have about a 64% chance of finding at least one "significant" result purely by chance (Family-Wise Error Rate, or FWER). Real experiment metrics are often correlated, so the exact number changes, but the core problem doesn't. Predeclare the primary decision metric; when you interpret a family of exploratory metrics, segments, or rubric axes, apply an appropriate correction such as Benjamini-Hochberg^[7] to control the False Discovery Rate (FDR).

The code below applies multiple testing corrections to experimental results. It takes a dictionary of metric names and their corresponding test statistics and p-values, and returns the adjusted significance outcomes.

Here resolution_rate survives Benjamini-Hochberg but not Bonferroni because another related metric is even stronger. latency_p95 and cost_per_session fail both corrections.

Sequential testing

Fixed-horizon A/B tests require choosing the sample size $N$ in advance and avoiding early stopping based on ordinary p-values. Checking results daily and stopping whenever they cross $p < 0.05$ inflates the false-positive rate. A registered sequential design^[8] permits planned interim looks by allocating error budget across those looks, allowing an early decision only when its boundary is crossed.

One approach uses a spending function such as O'Brien-Fleming to allocate total error budget across planned checkpoints. Early checks require stronger evidence. Exact critical values depend on the design, sidedness, information fraction, and analysis method; generate them with a statistical package and register them before traffic starts.

Repeatedly checking p-values without a sequential testing correction inflates your false positive rate. The exact inflation depends on the metric, test, and correlation between looks, which is why sequential methods matter.

The monitoring code below deliberately consumes precomputed boundaries instead of pretending to calculate a valid sequential test. It records four registered looks and stops only when the observed statistic exceeds that look's critical value:

Guardrail monitoring

Guardrails protect users and your production environment during exposure. Different signals require different actions. A synchronous personally identifiable information (PII) or unsafe-content detector can block one response immediately. A sampled judge safety rate, p95 latency, or average cost is an aggregate estimate with noise; it usually needs a defined window and persistence rule before you pause or roll back a treatment.

Setting appropriate thresholds requires balancing safety with experimental velocity. If thresholds are too tight, normal statistical noise will trigger false alarms and halt valid experiments. If they're too loose, users might be exposed to harmful model degradation before the system reacts.

Set guardrails from the baseline distribution you observed during stable periods, not from arbitrary fixed values. This reduces false rollbacks caused by normal day-to-day traffic swings.

This Python class monitors aggregate windows. A single breached window raises an alert; only a sustained breach reaches the predeclared action. In a real system, keep synchronous per-response blocking outside this aggregate loop.

The first window alerts without making an aggregate decision. The second sustained window reaches each registered action; an objective sustained error breach triggers rollback while latency and cost cause pause or review.

Multi-armed bandits

Standard A/B tests explore first (to find a winner) and exploit later (when you ship the winner to 100% of users). But what if the exploration phase is too costly? If a bad model variant causes users to churn, you want to stop sending traffic to it as quickly as possible.

Multi-armed bandits (MAB) dynamically adjust traffic allocation during the experiment based on real-time performance. Using algorithms such as Thompson Sampling, the system routes more traffic to variants that appear to be winning and less traffic to variants that appear weak. This minimizes "regret," which is the total penalty incurred by exposing users to suboptimal models. Bandits fit continuous optimization tasks (such as prompt tuning) where you have many variants and aim to automatically deprecate underperforming ones without waiting for a fixed-horizon A/B test.

But MABs introduce architectural complexity. They require a tightly coupled feedback loop where user actions (such as a thumbs up) immediately update the routing logic. In distributed systems with high latency or delayed rewards (for example, measuring 7-day retention), bandits can be challenging to implement correctly compared to static A/B test splits.

The following Python example illustrates a basic Beta-Bernoulli Thompson Sampling bandit. It assumes a binary reward signal such as click/no-click or accept/reject. If your reward is continuous or heavily delayed, the posterior update changes.

Biases that distort results

Evaluating generative AI introduces unique psychological and statistical biases that typically don't appear in standard software experiments.

Position bias

Users often prefer the first option shown, simply because it requires less effort to read. In pairwise comparisons (such as Side-by-Side evaluation or Reinforcement Learning from Human Feedback (RLHF) labeling), this bias can skew results significantly.

Solution

Always randomize the order of presentation. If showing two model outputs A and B, ensure A appears first for 50% of users and B appears first for the other 50%. Log the display order to control for this bias during analysis.

Running a head-to-head judge comparison with A always first can make order effects look like model quality. Swap positions and analyze order effects so presentation bias is measured and reduced rather than silently attributed to the model.

Novelty effect

Users initially engage more with new models or features just because they're different, not necessarily better. A new return-label feature might see a spike in usage on Day 1 that evaporates by Day 7.

Solution

Plan duration around known demand cycles and novelty risk. A full weekly cycle is a reasonable starting point when weekday and weekend traffic differ, but it isn't a universal stopping rule. If you want a burn-in window, define it before launch and exclude it consistently. Don't decide to drop early days after seeing a spike.

Simpson's paradox

A model can look better overall even while losing inside every major segment. This usually happens when the analyzed sample has different segment mixes across variants.

Both segment-level comparisons favor Model A, but the aggregate flips because Model B's analyzed sample contains far more mobile users (who have higher resolution rates overall). That's Simpson's paradox.

How to avoid it

Always segment results by pre-treatment user groups (for example, tenure, subscription tier, device type) and inspect the sample mix in each arm. Be especially careful with triggered analyses, missing-label subsets, or any filter that treatment can influence. Before declaring a winner, verify that the treatment doesn't severely degrade the experience for critical cohorts, even if the aggregate top-line metric improves.

The calculation below reproduces the aggregate reversal. Both pre-treatment cohorts favor Model A even though Model B's mobile-heavy analyzed sample wins after pooling:

LLM-as-judge for offline screening

LLM-as-Judge^[9] offers automated quality comparison to screen candidates before committing to a live A/B test. Full A/B tests with human evaluation are expensive and slow.

LLM judges are useful triage systems, not ground truth. They can cheaply filter obvious regressions in offline evaluation or shadow mode, but they also inherit position bias, verbosity bias, and judge-model bias^[9]. Use them to narrow the field before you consume valuable live traffic.

The asynchronous function below uses a strong LLM to evaluate two competing model outputs. It takes the original prompt and both responses as inputs, and returns a judgment ("A", "B", or "TIE") with a brief justification.

Key practices for LLM-as-judge

• Swap positions: To measure position bias, run comparisons in both A/B and B/A order.
• Use a capable judge candidate: Greater capability may help, but calibration against human labels decides whether the judge is usable.
• Calibrate against humans: Measure judge agreement on a held-out human-labeled set before you trust it for go/no-go decisions.
• Audit length bias: Slice wins by response length and refusal style to catch judges that reward verbosity instead of correctness.
• Don't replace A/B tests entirely: Use LLM-as-Judge to filter candidates, then validate the winning candidate with a real A/B test on production traffic.

Published judge evaluations report position and verbosity biases.^[9] Check whether your judge's wins correlate with display order or response length before trusting its verdict.

Online evaluation and the offline-online gap

Everything above the live A/B test (golden datasets, rubrics, offline judges) measures what your variant does on a frozen set you chose. Production measures what it does on traffic you did not choose. That difference is the offline-online gap, and closing it is the whole point of online evaluation. A variant can win 90% offline and still regress live because the real query mix is different, latency and cost shift under load, and rare safety failures only surface at the production distribution.

A common production pattern is to keep the judge running after launch instead of only before it. You sample a predeclared slice of live traffic sized for budget and detection needs, attach the request, retrieved context, tool calls, and output to a trace, then score that trace asynchronously with the same rubric you used offline. Because the eval is linked to a trace, a quality drop points you straight to the failing request rather than to an aggregate dashboard with no drill-down. This is the trace-linked online judge.

Keep two ideas separate, because production systems implement them differently:

An inline guardrail is a safety gate that must run before the response ships. An online judge is a measurement system that runs after, on a sample; when it is kept off the request path, it doesn't add user-facing latency. Confusing the two leads people to either slow every request with a judge call or to treat a slow async score as a real-time block.

Online judges also drift when the judge version, rubric, prompt, or traffic mix changes. The fix is the same calibration discipline you use offline, run continuously. Track judge-vs-human agreement on a rotating audit set, choose an acceptance threshold for your decision risk, log the full judge configuration on every score, and re-calibrate whenever that configuration changes.

This audit check uses an explicit local policy rather than a universal agreement cutoff:

Tooling

Managing LLM experiments at scale usually requires more than spreadsheets and ad hoc notebooks. Common choices include:

When selecting tools, prioritize integration with your existing ML infrastructure, immutable experiment definitions, trace linkage, and guardrail alerting. A useful toolchain connects offline evaluation to online experiments with shared metric definitions.

Practice: build a prompt arena

Your turn. Build a small "Prompt Arena" CLI tool that automates the workflow we walked through.

Requirements

1. Create a CSV file with 10 support questions (use the five golden questions from this article plus five more of your own).
2. Write two system prompts: a baseline and a treatment.
3. Send each question through both prompts using an API (OpenAI, Groq, or a local model via Ollama).
4. Use a judge candidate or human reviewers to score each response on Clarity, Accuracy, and Policy Citation, then record how that evaluator was calibrated.
5. Print a summary report: average scores per prompt, win rate, and which questions had the biggest gaps.

Check your understanding

• If Prompt A scores higher on Accuracy but lower on Policy Citation, which metric is more important for a support bot? (Answer: You must define that policy before seeing results. Factual policy correctness is a hard requirement; citation completeness may be primary or a guardrail depending on product policy.)
• You run 50 conversations and the treatment wins 60% of the time. Is that enough to ship? (Answer: Probably not. 50 conversations is far below the sample size needed for a 5-point lift at standard power. Run a power analysis.)
• Why is per-request randomization dangerous in a multi-turn support chat? (Answer: A user might see different tones, policies, or context states on each turn, which breaks the experiment and creates a terrible experience.)

Mastery check

This is the last article in Inference & Production Scale. You now have the full production loop: serve a model fleet, autoscale it, route and fall back across providers, watch cost and tokens, observe it in production, and decide what gets to ship through online evaluation. That last decision is the one this article owns, and you should be able to defend the full online-evaluation plan, not just quote a p-value.

Evaluation rubric

1. Design a rubric for a specific business use case and explain the difference between reference-based and reference-free evaluation.
2. Identify why a per-request randomization breaks a multi-turn conversation experiment.
3. Calculate a required sample size for a binary metric and explain why small samples lead to false positives.
4. Name three deployment patterns (shadow, canary, live A/B) and describe when each is appropriate.
5. Spot position bias, novelty effects, and Simpson's paradox in experimental results.

You should also be able to:

• Pick a randomization unit that avoids SUTVA violations. For collaborative products, that often means team, organization, document cluster, or switchback randomization instead of per-user randomization.
• Separate primary, secondary, and guardrail metrics. Resolution rate can be primary; safety rate, p95 latency, error rate, and cost per successful session are guardrails.
• Size the test before launch. A tiny golden dataset is useful screening evidence, not launch evidence.
• Choose shadow, canary, or live A/B testing based on risk and measurement goals.
• Add predefined circuit breakers for objective critical failures and persistence rules for noisy aggregate guardrails.
• Account for cache, cold-start, output-length, and routing bias before blaming the model.
• Choose interleaving for ranking comparisons where both systems can be shown in one mixed result list.
• Apply multiplicity corrections when many metrics, segments, or rubrics are tested at once.

Follow-up questions

How do you handle network effects in LLM A/B tests? Network effects break simple A/B tests when one user's treatment changes another user's experience. Shared AI-edited documents are a typical example: User A's improved edits can make User B happier even if User B is in the control group. Use cluster randomization at the team, organization, or document-cluster level. For short-lived effects, switchback experiments can also work.

Which common guardrails should an LLM product consider? Typical choices include p95/p99 latency, error rate, empty-response rate, safety violations, personally identifiable information (PII) leakage, and cost per successful session. Block objective per-response safety failures inline; pause or roll back exposure according to predefined persistence rules for aggregate signals.

How do you test multi-turn conversational effects? Randomize by session or conversation, not by request. Track conversation-level outcomes such as resolution rate, human handoff, sentiment trajectory, and retention after early turns. A bad first answer can poison every later turn.

When would you use a bandit instead of an A/B test? Use bandits when the goal is to optimize cumulative reward during exploration, such as continuous prompt tuning with fast feedback. Use classical A/B tests when you need clean inference, fixed exposure, long-term outcomes, or high-confidence launch evidence.

Common pitfalls

• Using vanity metrics instead of action metrics tied to product value.
• Randomizing every request in a multi-turn product.
• Skipping sample-size and minimum-detectable-effect planning.
• Ignoring cost per session and output-length growth.
• Treating every metric and segment as independent without multiplicity correction.
• Calling cold-start or cache-locality effects model regressions.
• Running without safety, reliability, and cost guardrails.
• Comparing judge outputs with A always first instead of swapping positions.
• Declaring victory from a tiny golden dataset without live validation.

If you can defend those decisions, you have closed out production scale. Next comes System Design Capstones, where you assemble serving, observability, guardrails, and online evaluation into one end-to-end design under interview conditions. The first capstone, a real-time content moderation system, leans directly on the guardrail and online-evaluation thinking from this article: it has to keep user-generated content safe at scale using LLMs and specialized classifiers.