A/B Testing for LLMs

GPU Serving & Autoscaling gave you a fleet that can survive live traffic. A/B testing decides whether a large language model (LLM), prompt, routing, or retrieval change should receive that traffic at all.

A/B testing (also called split testing) is a randomized experiment that divides users into two groups: a control group that receives the existing version of a feature, and a treatment group that gets the new version^{[1]Reference 1Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}. By comparing outcomes between groups, you can determine whether a change improves the feature or if observed differences are just noise.

A developer-doc assistant for an internal platform currently uses a simple system prompt: "Be helpful." Your team proposes a new prompt: "Be helpful, cite the relevant API docs or runbook, and use the repository and service context from the request." It sounds better, but how do you prove it?

Three engineers preferring one reply proves nothing. You need a real experiment. A clicked citation, accepted runbook, or question resolved without human escalation is directly observable; LLM answer quality often needs a rubric or reviewer, and generated wording can vary across runs. A 1% quality gain isn't worth an unacceptable latency or safety regression.

You'll take a developer-doc assistant through a real online experiment, from hypothesis to verdict, while learning metric selection, sample sizing, traffic routing, and biases that can fool you into shipping a regression.

Why can an LLM treatment win on quality but still fail the rollout decision?

Answer

Quality is only the primary metric. If latency, safety, reliability, or cost guardrails breach their thresholds, the treatment fails even when users or judges prefer its answers.

Why LLM A/B testing is different

Start with what LLM generation adds to ordinary online experimentation.

Observable events and scored quality

A button click is an observable binary event. Source-grounded answer quality isn't. It needs a defined rubric plus calibrated human or model judgments that turn each response into a reproducible label or score. Without that measurement contract, reviewers are comparing taste; with it, you can estimate preference or rubric-score differences and quantify uncertainty around the margin.

What makes an LLM A/B test different from a button-click A/B test?

Answer

LLM output quality is subjective, nondeterministic, and context-dependent. You need rubrics, judges or human labels, guardrails, and variance controls instead of only counting an objective click event.

Non-determinism

A/B tests already handle variable user outcomes: two developers assigned the same docs-search layout don't behave identically. LLM features add another variation source because the same prompt can yield different text as sampling, model serving, and context change. For offline comparisons, hold sampling and retrieval settings fixed and use deterministic decoding when the runtime supports it. If your stack exposes a seed, log it, but don't assume a seed alone makes live traffic reproducible. Otherwise, sampler or serving variation can obscure the treatment effect.

Freeze and log the serving contract in experiment metadata so offline gates and live arms stay comparable:

Live traffic still has residual noise (queueing, cache hits, co-tenants). The checklist doesn't eliminate that; it stops accidental offline/live mismatches from looking like a treatment effect.

The subjectivity problem

Unlike a button click (binary), LLM quality is subjective. Two engineers might disagree on whether a response is "helpful." Your experiments must move from "it looks good" to rigorous, quantifiable metrics. That's why modern LLM evaluation uses a layered approach rather than gut feel.

The cascade effect

Small changes in a system prompt can lead to outsized regressions on edge cases that aren't immediately visible. A prompt tweak that improves the median query might catastrophically fail your hardest 5% of cases. This makes guardrail metrics and golden datasets essential, not optional.

What exactly are you testing?

Don't limit your experiments to "Model A vs Model B." LLM A/B tests can evaluate many variables:

In our docs-assistant example, the most common test is a prompt duel: the same model with two different system prompts. Understanding what you're testing determines your metrics, sample size, and success criteria.

Why must you change only one experimental variable at a time when possible?

Answer

If prompt, model, retrieval, and temperature all change together, you cannot attribute the measured effect to one cause. Clean tests isolate the treatment so the verdict is actionable.

Three levels of LLM evaluation

Before you pick a metric, you need to know what family of evaluation fits your task. These three levels move from cheap and deterministic to flexible and expensive.

Level 1: Rule-based checks

For structured outputs or code generation, deterministic rules are the fastest and most reliable evaluators.

• Exact match and regex: Does the output contain a valid JSON object with the expected keys?
• Code execution: Does the generated Python script run and pass a set of unit tests?
• Format compliance: Does the response follow the requested template?

These are cheap, fast, and objective. Use them whenever you can. They are also reference-based: you compare the output against a known correct answer or schema.

Level 2: Semantic similarity

When the right answer can be phrased many ways, word-overlap metrics like BLEU and ROUGE often miss the point. They reward verbatim copying, which is terrible for creative or conversational tasks.

A better approach is embedding-based similarity.

• BERTScore^{[2]Reference 2BERTScore: Evaluating Text Generation with BERT.https://arxiv.org/abs/1904.09675}: Uses contextual embeddings to compare token-level similarity, capturing paraphrases.
• Cosine similarity: Measures how close two sentence embeddings are in vector space.

These estimate whether the model's answer has similar meaning to a reference answer, even if the wording differs. They don't prove factual correctness or policy compliance. Like Level 1, they are reference-based: you still need a golden answer to compare against.

Level 3: LLM-as-judge

For open-ended conversation, even embeddings fall short because there's no single correct phrasing. A common evaluation pattern is to use a separately calibrated judge model to grade candidate responses against a rubric and any required source context.

• G-Eval^{[3]Reference 3G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.https://arxiv.org/abs/2303.16634}: A judge LLM scores outputs on a rubric (for example, Clarity 1-5, Accuracy 1-5).
• Pairwise comparison: The judge sees two responses and picks the better one based on explicit criteria.

This can avoid a single golden wording, but factual or policy tasks still need source context in the rubric. It's scalable and flexible, but judges have their own biases. Later sections return to those judge risks.

Reference-based methods (Levels 1 and 2) work best when you know what the output should look like. Reference-free methods (Level 3) are useful for creative, conversational, or open-ended tasks where the "right" answer isn't fixed.

When should you prefer rule-based checks over LLM-as-judge?

Answer

Use rule-based checks when correctness is objectively testable, such as JSON validity, schema compliance, exact citations, or executable code. They are cheaper, faster, and less biased than judge models.

Worked example: the system prompt duel

A complete offline experiment makes the evaluation contract concrete. Compare Prompt A ("Be helpful.") against Prompt B ("Be helpful, cite the relevant API doc or runbook, and use the repository context.") on a small golden dataset.

Step 1: Build a golden dataset

A golden dataset is a curated benchmark of your hardest product cases. Unlike generic benchmarks (like MMLU or HumanEval), a golden dataset reflects your specific use case: your developers' most complex questions and your platform's highest-risk workflows.

For the docs assistant, start with five representative questions:

These are edge cases that matter. Passing them screens important failures, but it doesn't prove behavior on routine traffic or unseen edge cases. Before an online A/B test, require the variant to pass an offline eval gate appropriate to its risk. This prevents obvious failures from consuming user traffic.

What is the job of a golden dataset before online traffic sees a variant?

Answer

It screens candidates against curated high-value and hard cases. It does not prove docs impact, but it prevents obviously unsafe or low-quality variants from consuming live experimental traffic.

Step 2: Run the two prompts

Send each question through both system prompts. Record the outputs. For this offline comparison, request deterministic decoding where the selected runtime supports it and record all generation, retrieval, and model-version settings. Temperature zero reduces sampling variation in many runtimes; it doesn't by itself guarantee identical output across provider or backend changes.

In practice, you'd use an API client or a local model. Keep every parameter identical except the prompt itself, so you're measuring the prompt's effect, not sampler noise.

Step 3: Write the judge rubric

A rubric turns subjective quality into countable scores. Here's a simple three-axis rubric for our docs assistant:

You feed the judge LLM the question, the response, and this rubric, then ask it to score each axis from 1 to 5. The rubric turns the evaluation into a reviewable procedure: another engineer can run the same judge configuration and compare results, then measure agreement against held-out human labels.

Step 4: Score and calculate a winner

Suppose you run the five questions and get these average scores:

Prompt B wins on every axis. But notice that Prompt A never cited the source document, while Prompt B did so consistently. The rubric made that gap visible. Without the rubric, you might have glanced at the outputs and thought both were "fine."

In pairwise mode, you can also compute a win rate. If the judge compares A and B head-to-head on each question and declares B the winner 4 times, A the winner 0 times, and a tie 1 time, the win rate is:

$\text{Win Rate}_B = \frac{\text{Wins}_B + 0.5 \times \text{Ties}}{\text{Total Trials}} = \frac{4 + 0.5(1)}{5} = 0.90$

A 90% win rate over five examples is an encouraging screening signal on this curated set, not strong launch evidence. You would still need broader offline checks and a live A/B test to determine user impact, while this small duel can filter out obvious losers before you risk real traffic.

A golden dataset plus a clear rubric turns "looks good" into recorded screening evidence. Use an offline gate before live exposure whenever the treatment can change answer quality or safety.

Why is a 90% offline judge win rate not enough to ship by itself?

Answer

It only measures the curated offline set and the judge's rubric. Live users can react differently, traffic mix can shift, latency and cost can regress, and rare safety failures may appear only under the live distribution.

Metrics that matter in production

Offline rubrics are useful, but production experiments need metrics that reflect real value and system health. Successful tests use a taxonomy of metrics organized into three tiers based on how they're measured.

Keep latency terms straight, and document the formulas your tooling uses. TTFT is the delay until the first token arrives. ITL (also called TBT, time between tokens) is each gap between consecutive output tokens after the first. TPOT is the mean of those gaps for a request (or decode time after the first token divided by output tokens minus one). They are not synonyms: a healthy mean TPOT can hide a few long ITL spikes that still make streaming feel broken. Mixing those up makes latency regressions hard to diagnose because each metric points to a different bottleneck.

When latency is a guardrail, name the surface: TTFT, p95 ITL, mean TPOT, or end-to-end response time. A single generic "p95 latency" label hides whether prefill, streaming jitter, or mean decode pacing regressed. When the experimental variable is serving infrastructure (quantization, speculative decoding, batching knobs), prefer goodput (or dual TTFT + decode attainment) over one blended latency number so capacity wins aren't mistaken for quality wins.

Within each experiment, further categorize by priority:

1. Primary metrics: Direct business value. For a docs assistant, this could be "question resolved without human escalation" or developer satisfaction score.
2. Guardrail metrics: Predefined constraints. For example, a sustained TTFT or p95 ITL breach, mean TPOT regression, or statistically credible safety degradation can block promotion regardless of quality gains; exact policy thresholds depend on your baseline and SLO.
3. Secondary metrics: Debugging signals that explain why the primary metric moved, such as whether accepted code was shorter, more complete, or easier to review.

For LLM documents, cost belongs in the same dashboard as quality and latency. A variant can "win" by generating longer answers, invoking more tools, or consuming more retrieved context. Track cost per response, cost per successful session, and output-token growth as guardrails, not as an after-the-fact finance metric.

What is the difference between a primary metric and a guardrail metric?

Answer

The primary metric is what the experiment tries to improve, such as resolution rate. A guardrail is a constraint that must not degrade past a threshold, such as safety rate, p95 latency, error rate, or cost per successful session.

This Python dictionary categorizes typical metrics for evaluating our docs assistant, then applies a simple promotion policy to observed deltas. A positive primary result isn't enough when a predefined guardrail fails.

How many threads do you need?

Detecting a small improvement in LLM quality is like hearing a whisper in a noisy room. You need to listen longer (collect more data) to be sure it wasn't just random noise. If your metric has high variance (like token usage) or low signal (like thumbs up/down), you need a larger sample size to distinguish signal from noise.

Calculate the required sample size before starting so the test has statistical power (the probability of detecting a real effect).

A numeric warm-up

Suppose your docs assistant currently resolves 60% of threads on its own (the baseline rate). A five-point improvement to 65% is the smallest lift worth launching. How many threads do you need in each group to design a test with useful power for that target?

With a standard 5% false-positive rate and 80% power, the normal approximation gives roughly 1,471 threads per arm, or about 2,942 total. That's a lot for a low-traffic internal tool, which is why offline screening with a golden dataset matters. It lets you kill bad variants before they consume live traffic.

Notice what happens if you aim smaller. Detecting a 2-point lift (from 60% to 62%) balloons the requirement to over 18,600 total threads. Smaller effects require quadratically more samples.

The formula

For binary metrics (like "resolved" vs "not resolved"), a common two-sided normal approximation for a 50/50 test is:

$n_{\text{per arm}} \approx \frac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{\beta}\sqrt{p_A(1-p_A) + p_B(1-p_B)}\right)^2}{(p_B - p_A)^2}$

Here, $p_A$ is the baseline rate, $p_B = p_A + \delta$ is the treatment rate implied by your minimum detectable effect (MDE), and $\bar{p} = (p_A + p_B)/2$. The $z$ values come from the standard normal distribution: $z_{\alpha/2}$ controls false positives and $z_\beta$ controls false negatives.

Python calculator

This function computes the necessary sample size for a binary metric. It takes the baseline conversion rate, the minimum detectable effect, alpha, and power as inputs, and returns the total required sample size using the standard normal approximation.

Use this calculator before launch, not after results arrive. For the docs-assistant example, a 5-point lift needs thousands of threads, while a 2-point lift needs far more.

For judge scores, latency, or token counts, estimate variance from historical logs or a pilot run and use the matching power analysis for continuous metrics. Don't reuse a binary-proportion shortcut for everything.

Why does a 2-point lift need far more traffic than a 5-point lift?

Answer

Sample size grows roughly with the inverse square of the effect size. Smaller effects are harder to distinguish from noise, so they need many more observations.

Routing users to the right variant

The randomization unit determines how traffic and users are divided between the control and treatment groups. It affects both the statistical validity of the experiment and the consistency of the user experience.

For chat-like assistants, randomizing per user or per conversation is usually the right default. Pin that assignment for the full thread so follow-up turns, tool calls, and regenerations all hit the same variant. This keeps context and tone consistent, and it reduces within-session contamination where one bad turn changes how the user behaves on later turns.

The no-interference part of SUTVA (Stable Unit Treatment Value Assumption) can fail in collaborative features or shared documents because one user's assignment can affect another user's outcome. SUTVA also requires a well-defined treatment: don't hide materially different variants behind the same experiment label. When users influence each other, you often need cluster-based randomization rather than simple per-user hashing.

Why is per-request randomization usually wrong for multi-turn chat?

Answer

A conversation can carry prompt state, retrieved context, user expectations, and prior model behavior across turns. Switching variants mid-thread contaminates both UX and measurement.

Session pinning

Stable assignment sounds simple until you ship a multi-turn assistant. In practice, you need a deterministic routing key (for example user_id or conversation_id), persistent storage of the assigned variant, and telemetry that logs that assignment on every turn. If a conversation starts on Variant B and the second turn accidentally lands on Variant A, you haven't just damaged UX. You've invalidated the experiment because prompt state, retrieved context, and prior model behavior now leak across variants.

This deterministic router assigns at the conversation level. Each turn of one docs-assistant thread gets the same prompt variant, and the logged assignment can travel with every trace:

Shadow, canary, and live A/B tests

Different rollout patterns answer different questions:

For high-risk changes, a common escalation sequence is offline evaluation, then shadow, then canary, then a user-facing A/B test. A low-risk copy adjustment may not need every step; record the reason for skipping one.

What question does shadow traffic answer that a live A/B test answers differently?

Answer

Shadow traffic answers whether the variant can run safely and what it would have produced without showing it to users. A live A/B test measures real user behavior and product impact.

Cache locality and cold-start bias

LLM serving is stateful in ways normal web experiments aren't. Prefix caches, KV cache pages, and warm GPU workers strongly influence TTFT and throughput^{[5]Reference 5Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}. A shadow deployment doubles inference work, and a low-traffic canary can look artificially slow because it misses warm-cache reuse or lands on cold replicas. Measure warm and cold latency separately, and make sure the treatment has enough steady traffic to stay warm before you call a latency regression "real."

Interleaving: a blind taste test for ranking

For ranking developer-doc results, standard A/B tests compare outcomes across separate traffic groups. Interleaving is a within-query design^{[6]Reference 6Large-scale Validation and Analysis of Interleaved Search Evaluation.https://doi.org/10.1145/2094072.2094078}: it mixes candidates from both rankers into one list and attributes clicks back to each source. In the large-scale search experiments reported by Chapelle et al., interleaving was substantially more sensitive than A/B comparison for detecting ranking differences. That advantage is task-specific: it applies when both systems can safely contribute items to one ranking, not to two different free-form chatbot responses.

In practice, the system mixes results from both models and measures user preference by which results get clicked. A concrete example for developer-doc search:

Because interleaving compares rankers on the same query for the same user, it controls much of the query and user-intent variation within that trial. That can make it more sensitive than a standard A/B test for ranking problems.

Why can interleaving need fewer impressions than a standard A/B test for ranking?

Answer

Both systems are evaluated on the same query for the same user in the same result list, so user-intent and query-difficulty variance are mostly controlled within the trial.

Team draft interleaving

The "Team Draft" method builds a balanced mix of results. Imagine two team captains (Model A and Model B) picking their best remaining unseen result. When both teams have contributed the same number of items, choose the next captain randomly. Otherwise, the team with fewer items picks next.

This implementation follows that Team Draft rule. It accepts two ranked lists of results as inputs and outputs a single interleaved list along with the items attributed to each model. If one ranker has no unseen item left, the other ranker keeps contributing until it also exhausts or the list reaches k; one short or duplicate-heavy list must not truncate valid results from the other.

What does Team Draft interleaving need to log besides the final mixed list?

Answer

It must log which source system owns each displayed item and which items received clicks. Without ownership attribution, clicks cannot be converted into votes for Model A or Model B.

Statistical rigor

Multiple testing correction

If you searched through 20 independent secondary metrics at $\alpha=0.05$, you would have about a 64% chance of finding at least one "significant" result purely by chance (Family-Wise Error Rate, or FWER). Real experiment metrics are often correlated, so the exact number changes, but the core problem doesn't. Predeclare the primary decision metric; when you interpret a family of exploratory metrics, segments, or rubric axes, apply an appropriate correction such as Benjamini-Hochberg^{[7]Reference 7Controlling the false discovery rate: a practical and powerful approach to multiple testing.https://doi.org/10.1111/j.2517-6161.1995.tb02031.x} to control the False Discovery Rate (FDR).

This example applies multiple testing corrections to experimental results. It takes a dictionary of metric names and their corresponding test statistics and p-values, and returns the adjusted significance outcomes.

Here resolution_rate survives Benjamini-Hochberg but not Bonferroni because another related metric is even stronger. latency_p95 and cost_per_session fail both corrections.

Why does tracking twenty metrics make a single p < 0.05 result less trustworthy?

Answer

More simultaneous tests create more chances to find a false positive by luck. Multiple-testing correction controls that false-discovery risk before you declare a winner.

Sequential testing

Fixed-horizon A/B tests require choosing the sample size $N$ in advance and avoiding early stopping based on ordinary p-values. Checking results daily and stopping whenever they cross $p < 0.05$ inflates the false-positive rate. A registered sequential design^{[8]Reference 8Peeking at A/B Tests: Why it matters, and what to do about it.https://doi.org/10.1145/3097983.3097992} permits planned interim looks by allocating error budget across those looks, allowing an early decision only when its boundary is crossed.

One approach uses a spending function such as O'Brien-Fleming to allocate total error budget across planned checkpoints. Early checks require stronger evidence. Exact critical values depend on the design, sidedness, information fraction, and analysis method; generate them with a statistical package and register them before traffic starts.

Repeatedly checking p-values without a sequential testing correction inflates your false positive rate. The exact inflation depends on the metric, test, and correlation between looks, which is why sequential methods matter.

The monitoring code below deliberately consumes precomputed boundaries instead of pretending to calculate a valid sequential test. It records four registered looks and stops only when the observed statistic exceeds that look's critical value:

Why is daily p-value peeking invalid without a sequential design?

Answer

Each look spends some false-positive risk. If you repeatedly check and stop when p < 0.05, the overall chance of a false positive becomes larger than 5%.

Guardrail monitoring

Guardrails protect users and your serving environment during exposure. Different signals require different actions. An inline personally identifiable information (PII) or unsafe-content detector can block one response immediately. Sampled judge safety rate, p95 latency, or average cost is an aggregate estimate with noise; it usually needs a defined window and persistence rule before you pause or roll back a treatment. If a required guardrail is missing or non-finite, the window is inconclusive and exposure pauses; the missing value isn't a threshold pass.

Setting appropriate thresholds requires balancing safety with experimental velocity. If thresholds are too tight, normal statistical noise will trigger false alarms and halt valid experiments. If they're too loose, users might be exposed to harmful model degradation before the system reacts.

Set guardrails from the baseline distribution you observed during stable periods, not from arbitrary fixed values. This reduces false rollbacks caused by normal day-to-day traffic swings.

This Python class monitors aggregate windows. A single breached window raises an alert; only a sustained breach reaches the predeclared action. In a real system, keep synchronous per-response blocking outside this aggregate loop.

One complete window alerts without making an aggregate decision. A second sustained window reaches each registered action; an objective sustained error breach triggers rollback while latency and cost cause pause or review. In the final window, required cost telemetry is absent, so the result is inconclusive and exposure pauses immediately.

Why should guardrail thresholds come from baseline distributions instead of arbitrary round numbers?

Answer

Baseline distributions reveal normal traffic variance. Thresholds set from stable historical behavior reduce false rollbacks while still catching real safety, latency, error, or cost regressions.

Multi-armed bandits

Standard A/B tests explore first (to find a winner) and exploit later (when you ship the winner to 100% of users). But what if the exploration phase is too costly? If a bad model variant causes users to churn, you want to stop sending traffic to it as quickly as possible.

Multi-armed bandits (MAB) dynamically adjust traffic allocation during the experiment based on real-time performance. Using algorithms such as Thompson Sampling, the system routes more traffic to variants that appear to be winning and less traffic to variants that appear weak. This minimizes "regret," which is the total penalty incurred by exposing users to suboptimal models. Bandits fit continuous optimization tasks (such as prompt tuning) where you have many variants and aim to automatically deprecate underperforming ones without waiting for a fixed-horizon A/B test.

But MABs introduce architectural complexity. They require a tightly coupled feedback loop where user actions (such as a thumbs up) immediately update the routing logic. In distributed systems with high latency or delayed rewards (for example, measuring 7-day retention), bandits can be challenging to implement correctly compared to static A/B test splits.

This Python example implements a basic Beta-Bernoulli Thompson Sampling bandit. It assumes a binary reward signal such as click/no-click or accept/reject. If your reward is continuous or heavily delayed, the posterior update changes.

When should you prefer a classical A/B test over a bandit?

Answer

Prefer a classical A/B test when you need clean inference, fixed exposure, long-term outcomes, or regulatory/docs confidence. Bandits optimize reward during exploration but complicate causal analysis.

Biases that distort results

Generative outputs and filtered trace analysis make several familiar experiment biases easy to miss. None is exclusive to LLMs, but each can reverse an apparent winner if you ignore it.

Position bias

Human reviewers and judge models can favor the first option shown. Published LLM-judge evaluations report position bias^{[9]Reference 9Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}, and pairwise human review has the same measurement risk when presentation order isn't balanced.

Solution

Always randomize the order of presentation. If you're showing two model outputs A and B to reviewers, put A first for 50% of comparisons and B first for the other 50%. Log the display order to control for this bias during analysis.

Running a head-to-head judge comparison with A always first can make order effects look like model quality. Swap positions and analyze order effects so presentation bias is measured and reduced rather than silently attributed to the model.

Novelty effect

Users initially engage more with new models or features just because they're different, not necessarily better. A new citation-panel feature might see a spike in usage on Day 1 that evaporates by Day 7.

Solution

Plan duration around known demand cycles and novelty risk. A full weekly cycle is a reasonable starting point when weekday and weekend traffic differ, but it isn't a universal stopping rule. If you want a burn-in window, define it before launch and exclude it consistently. Don't decide to drop early days after seeing a spike.

Simpson's paradox

A model can look better overall even while losing inside every major segment. This usually happens when the analyzed sample has different segment mixes across variants.

Both segment-level comparisons favor Model A, but the aggregate flips because Model B's analyzed sample contains far more mobile users (who have higher resolution rates overall). That's Simpson's paradox.

How to avoid it

Always segment results by pre-treatment user groups (for example, tenure, subscription tier, device type) and inspect the sample mix in each arm. Be especially careful with triggered analyses, missing-label subsets, or any filter that treatment can influence. Before declaring a winner, verify that the treatment doesn't severely degrade the experience for critical cohorts, even if the aggregate top-line metric improves.

Why must segmentation use pre-treatment attributes?

Answer

If the treatment can change the segment or whether a row is included, the analysis becomes biased. Pre-treatment attributes like device, tier, or tenure existed before assignment and are safer for cohort checks.

The calculation below reproduces the aggregate reversal. Both pre-treatment cohorts favor Model A even though Model B's mobile-heavy analyzed sample wins after pooling:

LLM-as-judge for offline screening

LLM-as-judge^{[9]Reference 9Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} offers automated quality comparison to screen candidates before committing to a live A/B test. Full A/B tests with human evaluation are expensive and slow.

LLM judges are useful triage systems, not ground truth. They can cheaply filter obvious regressions in offline evaluation or shadow mode, but they also inherit position bias, verbosity bias, and judge-model bias^{[9]Reference 9Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}. Use them to narrow the field before you spend live traffic.

The asynchronous function below uses a strong LLM to evaluate two competing model outputs. It takes the original prompt and both responses as inputs, and returns a judgment ("A", "B", or "TIE") with a brief justification. Candidate outputs are untrusted data. A judge harness should isolate them from evaluator instructions with structured messages or a structured-input API, then test prompt-injection cases before using the judge for decisions.

Key practices for LLM-as-judge

• Swap positions: To measure position bias, run comparisons in both A/B and B/A order.
• Use a capable judge candidate: Greater capability may help, but calibration against human labels decides whether the judge is usable.
• Calibrate against humans: Measure judge agreement on a held-out human-labeled set before you trust it for go/no-go decisions.
• Audit length bias: Slice wins by response length and refusal style to catch judges that reward verbosity instead of correctness.
• Don't replace A/B tests entirely: Use LLM-as-Judge to filter candidates, then validate the winning candidate with a real A/B test on live traffic.

What two checks should you run before trusting an LLM judge for go/no-go screening?

Answer

Swap A/B order to measure position bias, and calibrate judge decisions against a held-out human-labeled set. Also slice by response length to catch verbosity bias.

Published judge evaluations report position and verbosity biases.^{[9]Reference 9Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} Check whether your judge's wins correlate with display order or response length before trusting its verdict.

Online evaluation and the offline-online gap

Everything above the live A/B test (golden datasets, rubrics, offline judges) measures what your variant does on a frozen set you chose. Production measures what it does on traffic you didn't choose. That difference is the offline-online gap, and closing that gap is the whole point of online evaluation. A variant can win 90% offline and still regress live because the real query mix is different, latency and cost shift under load, and rare safety failures only surface in production.

A common production pattern is to keep the judge running after launch instead of only before it. You sample a predeclared slice of live traffic sized for budget and detection needs, attach the request, retrieved context, tool calls, and output to a trace, then score that trace asynchronously with the same rubric you used offline. Because the eval is linked to a trace, a quality drop points you straight to the failing request rather than to an aggregate dashboard with no drill-down. This is the trace-linked online judge.

Keep two ideas separate, because production systems implement them differently:

An inline guardrail is a safety gate that must run before the response ships. An online judge is a measurement system that runs after, on a sample; when it's kept off the request path, it doesn't add user-facing latency. Confusing the two leads people to either slow every request with a judge call or to treat a slow async score as a real-time block.

Online judges also drift when the judge version, rubric, prompt, or traffic mix changes. Use the same calibration discipline you use offline, run continuously. Track judge-vs-human agreement on a rotating audit set, choose an acceptance threshold for your decision risk, log the full judge configuration on every score, and re-calibrate whenever that configuration changes.

This audit check uses an explicit local policy rather than a universal agreement cutoff:

Why run an LLM judge online after launch when you already screened the variant offline?

Answer

Offline screening only covers the curated set and the judge you froze. Online judging on sampled live traces catches the offline-online gap: shifted query mix, load-driven latency and cost regressions, rare safety failures, and silent drift in the served model or the judge itself.

Tooling

Managing LLM experiments at scale usually requires more than spreadsheets and ad hoc notebooks. Common choices include:

When selecting tools, prioritize integration with your existing ML infrastructure, immutable experiment definitions, trace linkage, and guardrail alerting. A useful toolchain connects offline evaluation to online experiments with shared metric definitions.