A/B Testing for LLMs

The candidate prompt wins four of five curated docs questions. A small live rollout (a canary) lifts resolution by 3.2%, but safety falls by 0.10 percentage points, p95 latency rises 410 ms, and cost rises $0.018 per session. Do you ship the winner? You hold it.

GPU Serving & Autoscaling gave you a fleet that can survive live traffic. Experiment Design and A/B Testing already taught you how to size a test, pin a randomization unit, and refuse to peek. Here, one large language model (LLM) prompt change has to earn live traffic without harming users or the serving budget.

An A/B test (also called split testing) randomly assigns eligible units to a control that keeps the current feature and a treatment that receives the change^{[1]Reference 1Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}. Comparing outcomes estimates whether the change helped or whether noise made it look good.

Our developer-doc assistant still uses "Be helpful." The candidate adds source citations plus repository and service context. Resolution without human escalation is the primary outcome; a clicked citation or accepted runbook is supporting evidence. Generated wording can vary, so quality needs a rubric, while latency, safety, and cost stay visible as separate constraints.

Why can an LLM treatment win on quality but still fail the rollout decision?

Answer

Quality is only the primary metric. If latency, safety, reliability, or cost guardrails breach their thresholds, the treatment fails even when users or judges prefer its answers.

Why LLM A/B testing is different

The canary result exposes the extra measurement work. A button click arrives as one observable event; an LLM response has to be judged in context, may differ on a repeat run, and can change the cost and latency of the request that produced it.

Observable events and scored quality

A button click is an observable binary event. Source-grounded answer quality isn't. It needs a defined rubric plus calibrated human or model judgments that turn each response into a reproducible label or score.

Without that measurement contract, reviewers are comparing taste. With it, you can estimate preference or rubric-score differences and quantify uncertainty around the margin.

Two engineers can disagree on whether a reply is "helpful." Move from "it looks good" to a recorded rubric score, then to resolution-without-escalation on live threads.

What makes an LLM A/B test different from a button-click A/B test?

Answer

LLM output quality is subjective, nondeterministic, and context-dependent. You need rubrics, judges or human labels, guardrails, and variance controls instead of only counting an objective click event.

Non-determinism

A/B tests already handle variable user outcomes: two developers assigned the same docs-search layout don't behave identically. LLM features add another variation source because the same prompt can yield different text as sampling, model serving, and context change.

For offline comparisons, hold sampling and retrieval settings fixed and use deterministic decoding when the runtime supports it. If your stack exposes a seed, log it, but don't assume a seed alone makes live traffic reproducible. Sampler or serving variation can otherwise obscure the treatment effect.

Freeze and log the serving contract in experiment metadata. That keeps offline gates and live arms comparable:

Live traffic still has residual noise from queueing, cache hits, and co-tenants. These settings don't eliminate that noise; they stop an offline/live mismatch from masquerading as a treatment effect. GPU Serving already showed why prefix cache and warm workers change TTFT, so an experiment that ignores those knobs can mistake a cold replica for a model regression.

Median wins can hide tail failures

A prompt tweak that improves the median query can still fail your hardest 5% of cases: the canary-analysis thread, the key-rotation runbook, or the v2 auth header. Guardrail metrics and a golden dataset of those edge cases belong before live traffic because an average can hide a dangerous tail.

Isolate the treatment

The canary tells you what happened, not which change caused it. Don't ship "Model A vs Model B" if the prompt, retriever, and temperature also moved. An LLM A/B test can change any of these, but isolate one treatment whenever possible:

In our docs-assistant example, the cleanest test is a prompt duel: the same model with two different system prompts. Once the treatment is named, the metric, sample size, and success rule have something precise to measure.

Why must you change only one experimental variable at a time when possible?

Answer

If prompt, model, retrieval, and temperature all change together, you can't attribute the measured effect to one cause. Clean tests isolate the treatment so the verdict is actionable.

Three levels of LLM evaluation

Before you pick a metric, you need to know what family of evaluation fits your task. LLM-as-a-Judge Evaluation covers judge design in depth. Here you only need the stack you'll use as an offline gate before live traffic: cheap deterministic checks first, then similarity to a reference, then a calibrated judge when wording isn't unique.

Level 1: Rule-based checks

For structured outputs or code generation, deterministic rules are the fastest and most reliable evaluators.

• Exact match and regex: Does the output contain a valid JSON object with the expected keys?
• Code execution: Does the generated Python script run and pass a set of unit tests?
• Format compliance: Does the response follow the requested template?

These are cheap, fast, and objective. Use them whenever you can. They are also reference-based: you compare the output against a known correct answer or schema.

Level 2: Semantic similarity

When the right answer can be phrased many ways, word-overlap metrics like BLEU and ROUGE often miss the point. They reward verbatim copying, which is terrible for creative or conversational tasks.

A better approach is embedding-based similarity.

• BERTScore^{[2]Reference 2BERTScore: Evaluating Text Generation with BERT.https://arxiv.org/abs/1904.09675}: Uses contextual embeddings to compare token-level similarity, capturing paraphrases.
• Cosine similarity: Measures how close two sentence embeddings are in vector space.

These estimate whether the model's answer has similar meaning to a reference answer, even if the wording differs. They don't prove factual correctness or policy compliance. Like Level 1, they are reference-based: you still need a golden answer to compare against.

Level 3: LLM-as-judge

For open-ended conversation, even embeddings fall short because there's no single correct phrasing. A common evaluation pattern is to use a separately calibrated judge model to grade candidate responses against a rubric and any required source context.

• G-Eval^{[3]Reference 3G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.https://arxiv.org/abs/2303.16634}: A judge LLM scores outputs on a rubric (for example, Clarity 1-5, Accuracy 1-5).
• Pairwise comparison: The judge sees two responses and picks the better one based on explicit criteria.

This can avoid a single golden wording, but factual or policy tasks still need source context in the rubric. It's scalable and flexible, but judges have their own biases. Later sections return to those risks, including position and verbosity effects reported in published LLM-as-judge evaluations^{[4]Reference 4Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}.

Reference-based methods (Levels 1 and 2) work best when you know what the output should look like. Reference-free methods (Level 3) are useful for creative, conversational, or open-ended tasks where the "right" answer isn't fixed.

When should you prefer rule-based checks over LLM-as-judge?

Answer

Use rule-based checks when correctness is objectively testable, such as JSON validity, schema compliance, exact citations, or executable code. They are cheaper, faster, and less biased than judge models.

Work the prompt duel offline

The rollout decision needs evidence before it needs traffic. Compare Prompt A ("Be helpful.") against Prompt B ("Be helpful, cite the relevant API doc or runbook, and use the repository context.") on a small golden dataset, then let each result earn the next check.

Choose cases that can fail

A golden dataset is a curated benchmark of your hardest product cases. Unlike generic benchmarks (like MMLU or HumanEval), it reflects your use case: developers' complex questions and your platform's highest-risk workflows.

For the docs assistant, start with five representative questions:

These cases expose failures that matter. Passing them screens important risks, but it doesn't prove behavior on routine traffic or unseen cases. Before an online A/B test, require the variant to pass an offline eval gate appropriate to its risk. That gate keeps obvious failures out of live traffic.

What is the job of a golden dataset before online traffic sees a variant?

Answer

It screens candidates against curated high-value and hard cases. It doesn't prove docs impact, but it prevents obviously unsafe or low-quality variants from consuming live experimental traffic.

Hold the run constant

Send each question through both system prompts and record the outputs. Request deterministic decoding where the selected runtime supports it, then record generation, retrieval, and model-version settings.

Temperature zero reduces sampling variation in many runtimes; it doesn't by itself guarantee identical output across provider or backend changes. Keep every parameter identical except the prompt so the comparison measures the prompt's effect rather than sampler noise.

Make quality scorable

A rubric turns subjective quality into countable scores. For this docs assistant, score three axes:

Give the judge LLM the question, the response, and this rubric, then ask it to score each axis from 1 to 5. Another engineer can rerun the same configuration, compare results, and measure agreement against held-out human labels.

Read a result as screening evidence

Suppose both prompts run on the five questions and produce these average scores:

Prompt B wins on every axis. Prompt A never cited the source document, while Prompt B did so consistently. The rubric makes that gap visible; without it, both replies might look "fine."

Pairwise judging gives a second view. If the judge compares A and B head-to-head and declares B the winner four times, A the winner zero times, and one tie, the win rate is:

$\text{Win Rate}_B = \frac{\text{Wins}_B + 0.5 \times \text{Ties}}{\text{Total Trials}} = \frac{4 + 0.5(1)}{5} = 0.90$

A 90% win rate over five examples is an encouraging screen on this curated set, not launch evidence. Broader offline checks and a live A/B test still determine user impact. This duel only filters obvious losers before they reach real traffic.

A golden dataset plus a clear rubric turns "looks good" into recorded screening evidence. Use an offline gate before live exposure whenever the treatment can change answer quality or safety.

Why is a 90% offline judge win rate not enough to ship by itself?

Answer

It only measures the curated offline set and the judge's rubric. Live users can react differently, traffic mix can shift, latency and cost can regress, and rare safety failures may appear only under the live distribution.

Name one primary outcome and guardrails

The offline rubric screens five golden questions, not user impact. Live traffic therefore needs one primary outcome plus a short list of constraints that can block promotion. First separate metrics by how you collect them:

Keep latency terms straight, and document the formulas your tooling uses. TTFT is the delay until the first token arrives. ITL (also called TBT, time between tokens) is each gap between consecutive output tokens after the first. TPOT is the mean of those gaps for a request (or decode time after the first token divided by output tokens minus one).

These aren't synonyms. A healthy mean TPOT can hide a few long ITL spikes that make streaming feel broken. Mixing them up makes latency regressions hard to diagnose because each metric points to a different bottleneck.

When latency is a guardrail, name the surface: TTFT, p95 ITL, mean TPOT, or end-to-end response time. A generic "p95 latency" label hides whether prefill, streaming jitter, or mean decode pacing regressed. When the experimental variable is serving infrastructure (quantization, speculative decoding, batching knobs), prefer goodput (or dual TTFT + decode attainment) over one blended latency number. Otherwise, a capacity win can look like a quality win.

Within each experiment, further categorize by priority:

1. Primary metrics: Direct business value. For a docs assistant, this could be "question resolved without human escalation" or developer satisfaction score.
2. Guardrail metrics: Predefined constraints. For example, a sustained TTFT or p95 ITL breach, mean TPOT regression, or statistically credible safety degradation can block promotion regardless of quality gains; exact policy thresholds depend on your baseline and SLO.
3. Secondary metrics: Debugging signals that explain why the primary metric moved, such as whether accepted code was shorter, more complete, or easier to review.

For LLM products, cost belongs in the same dashboard as quality and latency. A variant can "win" by generating longer answers, invoking more tools, or consuming more retrieved context. Track cost per response, cost per successful session, and output-token growth as guardrails rather than an after-the-fact finance metric.

What is the difference between a primary metric and a guardrail metric?

Answer

The primary metric is what the experiment tries to improve, such as resolution rate. A guardrail is a constraint that must not degrade past a threshold, such as safety rate, p95 latency, error rate, or cost per successful session.

This Python dictionary categorizes typical metrics for evaluating our docs assistant, then applies a simple promotion policy to observed deltas. A positive primary result isn't enough when a predefined guardrail fails.

Attributing user feedback across turns

In conversational LLM products, attribution starts by separating turn-level actions from session-level outcomes:

• Turn-level actions: Immediate explicit signals (copying a code snippet, clicking a cited documentation link, regenerating a reply, or giving a thumbs up/down to a specific assistant response) attach directly to that turn's request_id and prompt version.
• Session-level outcomes: End-to-end resolution (whether the user resolved their issue without filing an escalation ticket, or submitted a positive post-session rating) belongs to the full conversation_id.

Single-turn attribution can still mislead. If Turn 1 introduces subtle misinformation, a user might downvote only on Turn 3 after the suggested command fails. Assigning that signal purely to Turn 3's prompt misses the root cause.

Pinned session-level primary metrics evaluate the full conversational journey. Delayed feedback, such as an escalation ticket or offline survey response, must join back asynchronously to the session trace through immutable keys (conversation_id, assigned_variant, model_id) so outcomes don't disappear from the analysis.

Budget the traffic

The offline win doesn't tell you how much live traffic the prompt needs. A small improvement is easy to confuse with noise: high-variance metrics such as token usage and low-signal actions such as thumbs up/down need more observations.

You already planned power for an incident-assistant change in Experiment Design and A/B Testing. Calculate the required sample size before starting so this test has statistical power, the probability of detecting a real effect. Then add one LLM-specific check: hard queries may need more traffic than the pooled calculator suggests.

Budget for the smallest useful lift

Suppose the docs assistant currently resolves 60% of threads on its own. A five-point improvement to 65% is the smallest lift worth launching. How many threads does each group need for a test with useful power?

With a standard 5% false-positive rate and 80% power, the normal approximation gives 1,471 threads per arm, or 2,942 total. That's a lot for a low-traffic internal tool. Offline screening with a golden dataset matters because it removes bad variants before they consume live traffic.

Notice what happens when the target shrinks. Detecting a 3-point lift (60% to 63%) needs 8,258 total threads. A 2-point lift (60% to 62%) needs 18,672. Holding the other assumptions fixed, sample size grows roughly with the inverse square of the effect.

The binary calculator treats every thread as the same Bernoulli trial. Docs-assistant traffic isn't like that. "Which API limit applies to the batch export endpoint?" is a low-variance lookup; "Why did the deploy job fail after the canary analysis step?" is a high-variance debugging thread. If you size only on the pooled 60% rate, the hard slice can stay underpowered. Segment power by pre-treatment query type, or treat the overall $n$ as a floor and keep collecting until the hard slice is powered too.^{[1]Reference 1Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}

When a thread's recent resolution history predicts the outcome, CUPED (Controlled-experiment Using Pre-Experiment Data) can cut noise without adding traffic. You already saw that adjustment in Experiment Design; reuse it here when the pre-period covariate is available.^{[6]Reference 6Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED)https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf}

The formula

For binary metrics (like "resolved" vs "not resolved"), a common two-sided normal approximation for a 50/50 test is:

$n_{\text{per arm}} \approx \frac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{\beta}\sqrt{p_A(1-p_A) + p_B(1-p_B)}\right)^2}{(p_B - p_A)^2}$

Here, $p_A$ is the baseline rate, $p_B = p_A + \delta$ is the treatment rate implied by your minimum detectable effect (MDE), and $\bar{p} = (p_A + p_B)/2$. The $z$ values come from the standard normal distribution: $z_{\alpha/2}$ controls false positives and $z_\beta$ controls false negatives.

Python calculator

This function computes the necessary sample size for a binary metric. It takes the baseline conversion rate, the minimum detectable effect, alpha, and power as inputs, and returns the total required sample size using the standard normal approximation.

Use this calculator before launch, not after results arrive. For the docs-assistant example, a 5-point lift needs 2,942 threads, a 3-point lift needs 8,258, and a 2-point lift needs 18,672.

For judge scores, latency, or token counts, estimate variance from historical logs or a pilot run and use the matching power analysis for continuous metrics. Don't reuse a binary-proportion shortcut for everything.

Why does a 2-point lift need far more traffic than a 5-point lift?

Answer

Sample size grows roughly with the inverse square of the effect size. Smaller effects are harder to distinguish from noise, so they need many more observations.

Keep each thread in one arm

The traffic split is only useful when each experimental unit receives one coherent treatment. Choose the randomization unit before routing, because it controls both statistical validity and user experience.

Now name the estimand, the difference in the probability of resolving an eligible conversation under treatment versus control, measured over the predeclared outcome window. Assignment unit and analysis unit can differ, but the analysis must respect that relationship. If a user contributes several conversations or turns, account for that dependence instead of treating every row as independent evidence.

For chat-like assistants, randomizing per user or per conversation is usually the right default. Pin that assignment for the full thread so follow-up turns, tool calls, and regenerations all hit the same variant. Context and tone stay consistent, and within-session contamination is less likely to change how the user behaves after one bad turn.

The no-interference part of SUTVA (Stable Unit Treatment Value Assumption) can fail in collaborative features or shared documents because one user's assignment can affect another user's outcome. SUTVA also requires a well-defined treatment: don't hide materially different variants behind the same experiment label.

When users influence each other, simple per-user hashing no longer protects the estimate. Randomize at a cluster that contains the spillover, then analyze outcomes at a level that matches that design.

Why is per-request randomization usually wrong for multi-turn chat?

Answer

A conversation can carry prompt state, retrieved context, user expectations, and prior model behavior across turns. Switching variants mid-thread contaminates both UX and measurement.

Pin the conversation

Stable assignment sounds simple until you ship a multi-turn assistant. The router needs a deterministic key (for example user_id or conversation_id), persistent storage for the assigned variant, and telemetry that records the assignment on every turn.

If a conversation starts on Variant B and the second turn lands on Variant A, UX suffers first. The experiment also fails: prompt state, retrieved context, and prior model behavior now leak across arms.

This router assigns at conversation level. Every turn of one docs-assistant thread receives the same prompt variant, and the logged assignment travels with each trace. At a 10% treatment fraction, docs-thread-7 lands in treatment and docs-thread-48291 stays in control:

Choose the exposure step

Once assignment is coherent, choose how much of the user experience to expose. Each rollout pattern answers a different question:

For a high-risk change, the evidence path often moves from offline evaluation to shadow, canary, and finally a user-facing A/B test. A low-risk copy adjustment may skip a stage, but record what evidence makes that safe.

What question does shadow traffic answer that a live A/B test answers differently?

Answer

Shadow traffic answers whether the variant can run safely and what it would have produced without showing it to users. A live A/B test measures real user behavior and product impact.

Cache locality and cold-start bias

LLM serving is stateful in ways normal web experiments aren't. Prefix caches, KV cache pages, and warm GPU workers strongly influence TTFT and throughput^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}.

A shadow deployment doubles inference work. A low-traffic canary can also look slow because it misses warm-cache reuse or lands on cold replicas. Measure warm and cold latency separately, and give the treatment enough steady traffic to stay warm before calling a latency regression real.

When ranking can use interleaving

The prompt duel compares complete responses. Ranking has a different option: show one mixed result list to one query, then ask which ranker earned the clicks. Standard A/B tests compare separate traffic groups; interleaving is a within-query design^{[8]Reference 8Large-scale Validation and Analysis of Interleaved Search Evaluation.https://doi.org/10.1145/2094072.2094078} that mixes candidates from both rankers and attributes each item back to its source.

In the large-scale search experiments reported by Chapelle et al., interleaving needed far less query traffic than unpaired absolute metrics, the A/B-style design, to detect ranking differences. That advantage is task-specific: both systems must safely contribute items to one ranking. It doesn't transfer to two different free-form chatbot responses.

The system therefore mixes results from both rankers and measures preference by clicked source items. The mixed list below is the Team Draft output from the code that follows, using seed 7:

Because interleaving compares rankers on the same query for the same user, it controls much of the query and user-intent variation within that trial. That can make it more sensitive than a standard A/B test for ranking problems.

Why can interleaving need fewer impressions than a standard A/B test for ranking?

Answer

Both systems are evaluated on the same query for the same user in the same result list, so user-intent and query-difficulty variance are mostly controlled within the trial.

Team draft interleaving

The "Team Draft" method keeps the mix balanced. When both rankers have contributed the same number of items, choose the next source randomly. Otherwise, let the source with fewer items pick next.

The implementation accepts two ranked lists and returns one interleaved list plus source attribution. If one ranker has no unseen item left, the other keeps contributing until it also exhausts or the list reaches k; a short or duplicate-heavy list must not truncate valid results from the other.

What does Team Draft interleaving need to log besides the final mixed list?

Answer

It must log which source system owns each displayed item and which items received clicks. Without ownership attribution, clicks can't be converted into votes for Model A or Model B.

Protect the decision rule

Multiple testing correction

If you search 20 independent secondary metrics at $\alpha=0.05$, the chance of finding at least one "significant" result purely by chance is about 64% (Family-Wise Error Rate, or FWER). Real metrics are often correlated, so the exact number changes, but the selection problem remains.

Predeclare the primary decision metric. When you interpret a family of exploratory metrics, segments, or rubric axes, apply an appropriate correction such as Benjamini-Hochberg^{[9]Reference 9Controlling the false discovery rate: a practical and powerful approach to multiple testing.https://doi.org/10.1111/j.2517-6161.1995.tb02031.x} to control the False Discovery Rate (FDR).

This example applies multiple testing corrections to experimental results. It takes a dictionary of metric names and their corresponding test statistics and p-values, and returns the adjusted significance outcomes.

Here resolution_rate survives Benjamini-Hochberg but not Bonferroni because another related metric is even stronger. latency_p95 and cost_per_session fail both corrections.

Why does tracking twenty metrics make a single p < 0.05 result less trustworthy?

Answer

More simultaneous tests create more chances to find a false positive by luck. Multiple-testing correction controls that false-discovery risk before you declare a winner.

Sequential testing

Fixed-horizon A/B tests require choosing sample size $N$ in advance and avoiding early stopping based on ordinary p-values. Checking results daily and stopping whenever they cross $p < 0.05$ inflates the false-positive rate.

Sequential testing accounts for repeated looks with a preregistered error budget or always-valid inference. Johari and colleagues showed why ordinary peeking is invalid and how always-valid p-values make continuous monitoring part of the method.^{[10]Reference 10Peeking at A/B Tests: Why it matters, and what to do about it.https://doi.org/10.1145/3097983.3097992} You already simulated the inflation in Experiment Design: about 5% false wins at one planned look versus about 25% if you stop at the first crossing.

A registered group-sequential design is the other production pattern: it spends a planned error budget across a few interim looks. A spending function such as O'Brien-Fleming allocates most of that budget late, so early checks require stronger evidence. Exact critical values depend on the design, sidedness, information fraction, and analysis method; generate them with a statistical package and register them before traffic starts.

Repeatedly checking p-values without a sequential testing correction inflates your false positive rate. The exact inflation depends on the metric, test, and correlation between looks, which is why sequential methods matter.

The monitoring code below deliberately consumes precomputed boundaries instead of pretending to calculate a valid sequential test. It records four registered looks and stops only when the observed statistic exceeds that look's critical value. Treat the values as a schedule supplied by a registered design, not as universal cutoffs:

Why is daily p-value peeking invalid without a sequential design?

Answer

Each look spends some false-positive risk. If you repeatedly check and stop when p < 0.05, the overall chance of a false positive becomes larger than 5%.

Check assignment before lift

Before analyzing any metric lift, verify that observed traffic counts across arms match the planned allocation ratio, such as 50/50^{[1]Reference 1Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/[11]Reference 11Diagnosing Sample Ratio Mismatch in A/B Testing: A Taxonomy and Rules of Thumbhttps://doi.org/10.1145/3292500.3330722}. A statistically significant divergence is a Sample Ratio Mismatch (SRM).

An SRM is a count symptom, not a diagnosis. In LLM systems, it can expose hidden operational failures:

• Gateway timeouts and client drops: If a treatment prompt generates longer responses or suffers higher TTFT, requests may breach proxy deadlines (HTTP 504) or prompt users to close the tab before the first token renders. Dropped requests often fail before trace logging completes.
• GPU out-of-memory (OOM) crashes: Larger prompt context or higher KV cache consumption in the treatment arm can cause GPU workers to crash mid-decode.
• Asymmetric guardrail drops: Upstream moderation or safety filters that block treatment requests before assignment logging will artificially depress treatment counts.
• Session routing race conditions: Broken sticky hashing or unpersisted cookies cause multi-turn conversations to bounce between variants or drop attribution records.

To detect an SRM, run a Pearson chi-square ($\chi^2$) goodness-of-fit test comparing observed sample counts ($O_i$) to expected counts ($E_i = N \times r_i$):

$\chi^2 = \sum_{i \in \{\text{control}, \text{treatment}\}} \frac{(O_i - E_i)^2}{E_i}$

With 1 degree of freedom, test with a conservative significance threshold ($\alpha = 0.001$, critical $\chi^2 \approx 10.83$). If an SRM is detected ($p < 0.001$), the experiment is invalid. You can't trust an observed win or loss because selection bias has broken internal validity. Stop the experiment, find the infrastructure or logging defect, and restart.

Why must you check for Sample Ratio Mismatch before analyzing metric lifts?

Answer

An SRM signals that traffic was dropped, crashed, or misrouted unequally across variants. Any observed quality or business lift in an experiment with an SRM is invalidated by selection bias.

Turn guardrails into actions

Guardrails protect users and your serving environment during exposure, but each signal has its own response. An inline personally identifiable information (PII) or unsafe-content detector can block one response immediately.

Sampled judge safety rate, p95 latency, or average cost is an aggregate estimate with noise. Those signals usually need a defined window and persistence rule before you pause or roll back a treatment. If a required guardrail is missing or non-finite, the window is inconclusive and exposure pauses; missing telemetry isn't a threshold pass.

Compare the treatment to the concurrent control arm, not last week's dashboard. Query mix shifts. A canary that looks worse than last Tuesday may only be facing harder questions today.^{[1]Reference 1Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}

Thresholds balance safety with experimental velocity. Set them too tight and normal statistical noise triggers false alarms that halt valid experiments. Set them too loose and users may see harmful model degradation before the system reacts.

Set guardrails from the baseline distribution you observed during stable periods, not from arbitrary fixed values. This reduces false rollbacks caused by normal day-to-day traffic swings. Then evaluate the canary against the concurrent control, so a traffic-mix shift isn't mistaken for a model regression.

This Python class monitors aggregate windows. A single breached window raises an alert; only a sustained breach reaches the predeclared action. In a real system, keep synchronous per-response blocking outside this aggregate loop.

One breached window alerts without making an aggregate decision. A second sustained window reaches each registered action: an error breach triggers rollback, while latency and cost trigger pause or review. In the final window, required cost telemetry is absent, so the result is inconclusive and exposure pauses immediately.

Why should guardrail thresholds come from baseline distributions instead of arbitrary round numbers?

Answer

Baseline distributions reveal normal traffic variance. Thresholds set from stable historical behavior reduce false rollbacks while still catching real safety, latency, error, or cost regressions.

When allocation adapts: bandits

The fixed A/B split buys clean exposure and inference. It also keeps sending traffic to a weak arm while the test learns. If that exposure is costly, an adaptive design can trade some inferential simplicity for faster allocation.

Multi-armed bandits (MAB) adjust traffic during the experiment from observed performance. With Thompson Sampling, the router samples a plausible reward rate for each arm, then sends more traffic to arms that repeatedly look promising. The objective is lower regret, the cumulative penalty from exposing users to weaker variants.

That feedback loop changes the question. Bandits can suit continuous optimization with many prompt variants, but delayed rewards such as 7-day retention make rapid updates misleading. They also need routing, logging, and posterior updates to agree in real time, so a classical A/B test remains the safer choice when fixed exposure and clean causal analysis matter.

This Python example implements a basic Beta-Bernoulli Thompson Sampling bandit. It assumes a binary reward signal such as click/no-click or accept/reject. If your reward is continuous or heavily delayed, the posterior update changes.

When should you prefer a classical A/B test over a bandit?

Answer

Prefer a classical A/B test when you need clean inference, fixed exposure, long-term outcomes, or regulatory/docs confidence. Bandits optimize reward during exploration but complicate causal analysis.

Watch the path from exposure to verdict

The model isn't the only source of bias. Generative outputs, filtered traces, and changing traffic mixes can each turn a plausible result into a false winner. Follow the path from what users saw to what the analysis kept.

Position bias

Human reviewers and judge models can favor the first option shown. Published LLM-judge evaluations report position bias^{[4]Reference 4Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}, and pairwise human review has the same measurement risk when presentation order isn't balanced.

Measure position effects

Randomize presentation order. If you're showing two model outputs A and B to reviewers, put A first for 50% of comparisons and B first for the other 50%. Log display order so the analysis can measure the effect.

Running a head-to-head judge comparison with A always first can make order effects look like model quality. Swap positions and analyze order effects so presentation bias is measured and reduced rather than silently attributed to the model.

Novelty effect

Users initially engage more with new models or features just because they're different, not necessarily better. A new citation-panel feature might see a spike in usage on Day 1 that evaporates by Day 7.

Let novelty settle

Plan duration around known demand cycles and novelty risk. A full weekly cycle is a reasonable starting point when weekday and weekend traffic differ, but it isn't a universal stopping rule. If you want a burn-in window, define it before launch and exclude it consistently. Don't drop early days after seeing a spike.

When the mix creates Simpson's paradox

A model can look better overall even while losing inside every major segment. The reversal appears when the analyzed sample has different segment mixes across variants.

Both segment-level comparisons favor Model A, but the aggregate flips because Model B's analyzed sample contains far more mobile users (who have higher resolution rates overall). That's Simpson's paradox.

Check the mix before aggregating

Always segment results by pre-treatment user groups (for example, tenure, subscription tier, device type) and inspect the sample mix in each arm. Be especially careful with triggered analyses, missing-label subsets, or any filter that treatment can influence. Before declaring a winner, verify that the treatment doesn't severely degrade the experience for critical cohorts, even if the aggregate top-line metric improves.

Why must segmentation use pre-treatment attributes?

Answer

If the treatment can change the segment or whether a row is included, the analysis becomes biased. Pre-treatment attributes like device, tier, or tenure existed before assignment and are safer for cohort checks.

The calculation below reproduces the aggregate reversal. Both pre-treatment cohorts favor Model A even though Model B's mobile-heavy analyzed sample wins after pooling:

Use judges to screen, not declare truth

LLM-as-judge^{[4]Reference 4Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} offers automated quality comparison before a candidate reaches a live A/B test. Full A/B tests with human evaluation are expensive and slow, so a judge can narrow the field.

Treat that judge as a triage system, not ground truth. It can filter obvious regressions in offline evaluation or shadow mode, but it also inherits position, verbosity, and self-enhancement bias^{[4]Reference 4Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}. A pairwise judge may favor the first answer shown or reward a longer, repetitive answer that adds no information. Use it to decide what deserves live exposure, not to skip live evidence.

Candidate outputs are untrusted data. A judge harness should keep evaluator instructions in one channel and serialize the query plus both replies as JSON, then test prompt-injection cases before using the judge for decisions. The function below builds that boundary. It doesn't call a model; it proves the attack string stays inside the payload instead of being concatenated into the instruction text.

Calibrate before you trust a judge

Swap positions and run comparisons in both A/B and B/A order. A capable judge may help, but calibration against held-out human labels decides whether it's usable for a go/no-go screen.

Slice wins by response length and refusal style to catch a judge that rewards verbosity instead of correctness. Keep the live A/B test: judging filters candidates, while user-facing traffic validates impact.

What two checks should you run before trusting an LLM judge for go/no-go screening?

Answer

Swap A/B order to measure position bias, and calibrate judge decisions against a held-out human-labeled set. Also slice by response length to catch verbosity bias.

Published judge evaluations report position and verbosity biases.^{[4]Reference 4Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} Check whether your judge's wins correlate with display order or response length before trusting its verdict.

Close the offline-online gap

The five-question screen answered a narrow question: how did the prompts behave on cases you selected? Production asks a wider one: what happens on traffic you didn't select? That difference is the offline-online gap. A variant can win 90% offline and regress live when query mix changes, load shifts latency or cost, or rare safety failures appear only in production.

A common production pattern keeps the judge running after launch. Sample a predeclared slice of live traffic sized for budget and detection needs. Attach the request, retrieved context, tool calls, and output to a trace, then score that trace asynchronously with the same rubric used offline.

Trace linkage turns an aggregate quality drop into a debuggable request: you can inspect the failing context and output instead of staring at a total with no drill-down. This is the trace-linked online judge.

Keep two ideas separate, because production systems implement them differently:

An inline guardrail is a safety gate that must run before the response ships. An online judge is a measurement system that runs after, on a sample; when it's kept off the request path, it doesn't add user-facing latency. Confusing the two leads people to either slow every request with a judge call or to treat a slow async score as a real-time block.

Online judges also drift when the judge version, rubric, prompt, or traffic mix changes. Keep the offline calibration discipline running after launch. Track judge-vs-human agreement on a rotating audit set, choose an acceptance threshold for your decision risk, log the full judge configuration on every score, and recalibrate whenever that configuration changes.

This audit check uses an explicit local policy rather than a universal agreement cutoff:

Why run an LLM judge online after launch when you already screened the variant offline?

Answer

Offline screening only covers the curated set and the judge you froze. Online judging on sampled live traces catches the offline-online gap: shifted query mix, load-driven latency and cost regressions, rare safety failures, and silent drift in the served model or the judge itself.

Keep the evidence path intact

Once traffic is live, the experiment is only as trustworthy as the path from assignment to verdict. Product names change, but the evidence contract stays:

When you pick a tracker, eval runner, or trace UI, score it against that contract. A useful toolchain connects offline evaluation to online experiments with the same metric names.