Experiment Design and A/B Testing

An on-call engineer stares at a dashboard showing a critical payment service at 7% error rate. The service owner acknowledged the page two minutes ago. The incident assistant suggests an immediate canary rollback in clean prose. That reply can still cite the wrong runbook, take ten seconds to answer, or send the engineer to a human page five minutes later. Should the team ship a query rewrite?

In Decoding Algorithms, you controlled how retrieved runbook passages turned into an answer and logged the decoding policy. Here we focus on an incident assistant and ask a product question: does rewriting its retrieval query help engineers resolve outages? Keep generation and model weights fixed so the experiment isolates that retrieval change.

An A/B test, also called a randomized controlled experiment, assigns incoming incidents randomly to either the existing system (control) or the new system (treatment) and compares their outcomes. A retrieval update earns a production rollout only when rigorous measurement proves it helps without breaking on-call operations.

That measurement supports one of three clear decisions: ship the rewrite, keep investigating, or roll it back.

Test one change at a time

Start with the incoming request that both arms must answer:

The canary is at 7% errors and the owner acked. Can we roll it back?

Your existing pipeline retrieves runbook passages directly from the raw question and generates an answer. The control path keeps that exact behavior.

A candidate treatment first rewrites the request into a retrieval-focused query: canary rollback error-rate policy owner acknowledgement. It can't invent a rollback threshold that wasn't in the request or trusted context. It then sends the retrieved evidence through the same prompt, model, and decoder settings as control. Any measured difference belongs squarely to the rewrite policy, including its latency overhead and network errors.

Write a hypothesis that can fail:

Query rewriting will increase the resolved-incident rate while keeping grounded-answer audit failures, p95 latency, and human pages inside their budgets.

Here, p95 latency represents the 95th percentile of first-request completion times, recorded once per enrolled incident. It protects the slower tail rather than hiding it inside an average. Predeclare how timeouts enter the metric; silently dropping unfinished requests could make a failing treatment look faster.

Lock the decision rules before results tempt you to change them:

The brief specifies an MDE, or minimum detectable effect: the smallest lift the planned enrollment can detect with target statistical power. Keep that planning target separate from the later launch threshold.

The grounding rule asks whether the data rule out a regression as large as its budget. That's a non-inferiority question, which we'll calculate near the end. Speed and human pages remain point-estimate screens in this compact example, not statistical assurances of safety. A deployable brief needs uncertainty rules for those guardrails too; p95 uncertainty requires request-level data, not just the two reported percentiles.

Random assignment supports causal conclusions only when assignment, instrumentation, and stopping rules stay trustworthy. Practical online experiment systems make this pre-launch contract explicit.^{[1]Reference 1Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}

Assign an incident once

Why assign by incident rather than by message? An engineer might ask a follow-up after seeing the first answer. If the first message uses treatment and the follow-up uses control, the second outcome depends partly on the first answer. The unit has received both policies, so the contrast is no longer clean.

In statistical causal inference, this principle is formalized as the Stable Unit Treatment Value Assumption (SUTVA). SUTVA requires two conditions:

1. No interference between units: Unit $i$'s outcome depends only on unit $i$'s assigned treatment, unaffected by the assignment of unit $j$.
2. No hidden variations of treatment: The treatment policy assigned to unit $i$ is identical regardless of how unit $i$ received it.

When you randomize individual chat messages instead of the entire incident, SUTVA breaks immediately. The LLM's multi-turn conversation memory carries context across turns, exposing a single incident to a mixture of both variants.

SUTVA can fail across incidents through two main paths:

• Behavioral spillover: An on-call engineer sees a treatment recommendation that clarifies a tricky canary rollback procedure, and then uses that learned knowledge 20 minutes later on a control incident.
• Shared compute contention: If the treatment's query rewriter makes extra LLM inference calls or triggers heavy vector database scans, it can exhaust shared GPU worker pools, increasing latency for control incidents hosted on the same infrastructure.

When incidents cluster naturally inside engineering teams, randomize at the cluster level. Intracluster correlation inflates outcome variance by the design effect:

$$\text{DEFF} = 1 + (m - 1)\rho_{\text{ICC}}$$

Here $m$ represents average cluster size and $\rho_{\text{ICC}}$ is the intracluster correlation coefficient. More chat messages within one team can't replace independently randomized teams.^{[2]Reference 2Why Tenant-Randomized A/B Test is Challenging and Tenant-Pairing May Not Workhttps://www.microsoft.com/en-us/research/articles/why-tenant-randomized-a-b-test-is-challenging-and-tenant-pairing-may-not-work/}

A stable hash gives each enrolled incident the same arm on every request. Include an experiment name and version in the salt so a later experiment can get a fresh assignment. Lock the salt before outcomes arrive, keep incident IDs immutable, and don't let callers choose IDs to obtain a preferred arm. Hashing approximates random allocation here; it doesn't force exactly equal arm sizes.

If the same incident appears twice in the output, the assigned arm must match twice. In a service, persist the experiment version and arm on each event as well. Hashing makes the choice deterministic; logging makes that choice auditable.

Why should one incident keep the same experiment assignment across all follow-ups?

Answer

Switching variants between messages contaminates the treatment contrast and gives one incident's later behavior exposure to both policies. Assign the stable experimental unit once.

Define an outcome from events

The assistant shouldn't get credit merely because it produced an answer. Resolution requires a usable-answer mark by ten minutes after enrollment and no human page through that deadline. A missing mark counts as unresolved under this definition; broken telemetry is a measurement problem to investigate, not evidence of success.

Keep incidents whose treatment rewrite timed out in the treatment denominator. This intention-to-treat (ITT) comparison estimates the effect of assignment to the rewrite policy, failures included. Comparing only successful rewrites would select incidents using a treatment-affected event and answer a different, biased question. If 5% of treatment rewrites fail with timeouts, dropping them creates survivor bias by retaining only easier queries.

Wait until every enrolled incident has its full follow-up window. A nine-minute-old incident with no page isn't yet a ten-minute success. The rows below are already matured incident summaries, not individual chat messages; solved means a usable mark arrived within the locked window.

The next lab turns raw product events into that metric. It also records two guardrails: whether an audit says the answer is grounded in the retrieved runbook, and the response latency in milliseconds. Read the event rows before looking at the aggregate counts: which page timing should disqualify a usable answer?

The small rows teach the definition, not the launch result. The treatment row with escalation_after_min=18 still counts as resolved because its page arrives outside the locked ten-minute window. That makes the metric contract inspectable and deterministic.

If later pages matter, choose a longer window or track them separately before launch. The metric also needs enough incidents to distinguish an actual lift from chance variation. The next step is to turn these event-level decisions into an effect estimate.

Read lift from counts

After the planned window, start with a compact teaching fixture of 2,000 incidents per arm. The counts are small enough to check by hand. Before reading the table, predict which arm has the higher resolution rate and what could still make that apparent win misleading.

This fixture isn't powered for the locked MDE of +2.0 percentage points. The planning lab later shows you need about 9,246 incidents per arm at 80% power for that MDE.

Keep the roles separate: use the 2,000-row tables to practice lift, intervals, and launch gates; use the powered enrollment estimate when you design a real test for the brief.

Before subtracting the rates, name the target quantity. The estimand is the average effect of assignment to the rewrite policy on resolution among eligible incidents, with each incident weighted once. It includes rewrite failures and follow-up behavior. Random assignment lets us estimate that target by subtracting the two assigned-arm rates, under the no-interference and measurement assumptions already stated.

For the grounding counts below, assume every enrolled incident's answer receives the same blinded audit. If only a sample is audited, use the audited denominator and a predeclared sampling design rather than dividing audit failures by all incidents.

The absolute lift is the treatment rate minus the control rate. It answers a practical question: how many additional incidents resolve per 100 enrolled incidents?

$$\widehat{\Delta} = \hat{p}_{treatment} - \hat{p}_{control} = \frac{840}{2000} - \frac{760}{2000} = 0.04$$

The treatment resolves 4.0 additional incidents per 100 enrolled incidents. That's a 4.0 percentage point lift.

The relative lift compares that absolute change with the original baseline. It answers a different question: how large is the change compared with the control rate?

$$\frac{0.04}{0.38} \approx 0.105$$

That's a 10.5 percent relative lift. Name which one you report; saying "up 10.5 points" would be wrong because points describe the absolute difference here.

Treatment resolves 840 of 2,000 incidents and control resolves 760 of 2,000. What's the clearest user-facing interpretation of absolute lift?

Answer

The rewrite resolved 4.0 additional incidents per 100 enrolled incidents: 42.0% instead of 38.0%.

Put uncertainty around the lift

If you reran the experiment with different incidents, the counts wouldn't be identical. The observed lift is one draw from a larger process, so it needs a measure of how much it could move.

In Hypothesis Tests, Intervals, and pass@k you put a confidence interval around a paired lift. Here the two arms contain independent incidents, so the standard error adds their two Bernoulli variances. For reasonably large binary-outcome arms, a normal-approximation interval is a useful first calculation.

For each arm, $\hat{p}$ is its observed resolution rate and $n$ is its number of enrolled incidents. The estimated standard error of the difference is:

$$SE(\widehat{\Delta}) = \sqrt{ \frac{\hat{p}_{control}(1 - \hat{p}_{control})}{n_{control}} + \frac{\hat{p}_{treatment}(1 - \hat{p}_{treatment})}{n_{treatment}} }$$

A rough 95 percent interval is $\widehat{\Delta} \pm 1.96 \times SE$. The number 1.96 is the standard-normal cutoff that leaves about 2.5 percent in each tail.

Across repeated experiments satisfying these assumptions, this procedure aims to cover the true effect about 95% of the time. The particular interval we calculate isn't a 95% probability statement about a fixed true effect, and it doesn't cover a different population automatically.

The estimate is +4.0 points and this approximation gives an interval of about +1.0 to +7.0 points. Because the lower bound is above zero, this planned analysis gives evidence for a positive treatment effect, assuming assignment and instrumentation are sound.

It doesn't prove every future rollout will gain four points. The interval answers how much the estimate could vary; it still leaves the planning question open: was 2,000 per arm enough to detect the +2.0 point win you said you cared about?

For rare outcomes, heavily clustered incidents, many simultaneous comparisons, or business-critical launches, choose the inference method with a statistician or a mature experimentation platform before running the test.

More rows aren't always more evidence

Suppose a logging join copies every incident outcome onto four message rows. Both arm rates stay unchanged, so the lift still looks right. But treating those copies as independent observations divides the estimated standard error by two. No new incidents were enrolled; the extra precision is fictional.

Duplicating rows changes the denominator in the formula, not the amount of independent evidence. Aggregate once per randomized incident before computing this interval.

Real within-team correlation is usually less extreme than identical copies, but the same independence issue applies. If teams are randomized, use a team-aware estimator and uncertainty calculation, such as cluster-robust inference or resampling whole teams. Choosing a cluster method doesn't itself fix interference between clusters. The assignment unit, estimand, and analysis must agree.^{[2]Reference 2Why Tenant-Randomized A/B Test is Challenging and Tenant-Pairing May Not Workhttps://www.microsoft.com/en-us/research/articles/why-tenant-randomized-a-b-test-is-challenging-and-tenant-pairing-may-not-work/}

Plan for a meaningful win

A test with too little traffic may return "uncertain" even when the treatment helps. Before launching, declare a minimum detectable effect (MDE): a true effect size at which you want a specified probability of rejecting the no-effect null. Smaller effects can still be detected, and an effect at the MDE can still be missed.

Power planning balances three competing parameters:

1. False positive rate (Type I error, $\alpha$): The budget for false alarms when there's truly zero effect. Standard practice sets $\alpha = 0.05$ two-sided ($z_{1-\alpha/2} \approx 1.96$).
2. False negative rate (Type II error, $\beta$): The risk of missing a real effect of size MDE. Standard practice sets $\beta = 0.20$.
3. Statistical power ($1 - \beta$): The probability of successfully rejecting the null hypothesis when the true lift equals the MDE. Standard practice targets 80% ($z_{1-\beta} \approx 0.8416$).

Hold the observed rates at 38 and 42 percent, then watch what sample size does to the interval:

With 200 or 500 incidents per arm, a good-looking +4.0 point estimate is still compatible with no improvement. At 2,000 per arm, this observed +4.0 pp effect's interval finally excludes zero in the teaching fixture.

That isn't the same as being powered to detect the locked design MDE of +2.0 pp, which needs about 9,246 incidents per arm below. More enrollment narrows uncertainty; it doesn't make treatment better by itself.

The sample size required per arm follows an inverse square relationship with the effect size $\delta$:

$$n \approx \frac{2\sigma^2(z_{1-\alpha/2} + z_{1-\beta})^2}{\delta^2}$$

For binary proportions near baseline rate $p$, outcome variance is $\sigma^2 = p(1 - p)$:

$$n \approx \frac{2p(1-p)(z_{1-\alpha/2} + z_{power})^2}{\delta^2}$$

Notice the $\delta^2$ in the denominator: halving the MDE from +2.0 pp to +1.0 pp quadruples the required sample size.

This formula is for planning, not post-result storytelling. Actual platform planning may account for unequal allocation, repeated incidents, variance reduction, multiple metrics, or sequential analysis.

For comparison, the fuller independent-proportions calculation at baseline 0.38, true lift 0.02, equal allocation, and two-sided alpha 0.05 gives about 79.6% power at 9,246 per arm and rounds up to 9,336 per arm for 80%. That small difference explains why the shortcut is useful for sizing the project but shouldn't be mistaken for the final analysis specification.^{[3]Reference 3power_proportions_2indephttps://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.power_proportions_2indep.html}

An 80% power target isn't an 80% chance of satisfying the whole launch gate. If the true lift is exactly +2.0 points, a roughly symmetric estimate falls above the +2.0-point observed-lift threshold only about half the time. Requiring guardrails to pass can reduce launch probability further. Power the actual decision you intend to make, not just its easiest condition.

Use the estimate to decide how much traffic to enroll before the first outcome arrives. Predeclare a minimum calendar duration too when weekdays, on-call rotations, or novelty effects matter. Hitting the sample count during one unusual shift doesn't establish performance across those conditions.

None of that math is trustworthy if the two arms didn't actually receive the traffic you planned.

Check traffic before outcomes

A sample ratio mismatch (SRM) means observed assignment counts disagree suspiciously with the intended split. If the design says 50/50 but your data has 2,350 treatment incidents and 1,650 control incidents, pause before reading resolution lift. Assignment code, event logging, eligibility filters, or bot removal may differ by arm.

This lab computes a chi-square diagnostic for a planned 50/50 allocation. A one-degree-of-freedom value above 10.83 is a deliberately strict alert threshold corresponding to a very small tail probability, about 0.1 percent.

The 0.001 threshold is this lab's choice, not a universal standard. Microsoft's documented platform uses an even stricter 0.0005 threshold. Such a check diagnoses allocation or data-quality problems; it isn't a test of whether treatment helps.^{[4]Reference 4Diagnosing Sample Ratio Mismatch in A/B Testinghttps://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/}

An SRM alert doesn't identify the bug. It says your comparison hasn't earned trust yet. Stop and diagnose the event path before making a lift claim.

If the first look at lift looks unbelievable, treat instrumentation as the leading hypothesis until you can replay the events.^{[1]Reference 1Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}

Check distinct enrolled incident IDs, not message counts. Treatment could legitimately change the number of follow-ups, so expecting a 50/50 split of messages would test the wrong denominator. A clean SRM result is necessary evidence about traffic, not proof that every outcome field is correct.

Don't stop the first time noise looks good

Suppose treatment truly has no effect: both arms resolve 40 percent of incidents. One planned two-sided 5% test should falsely detect a difference about 5% of the time. That includes apparent benefits and apparent harms. It isn't a 5% rate of positive wins alone. Predict what happens when the same cutoff is checked 20 times and the first crossing ends the test.

Inspecting the ordinary interval after every batch gives noise many chances to cross the threshold. By the Law of the Iterated Logarithm, the cumulative path of zero-effect Brownian noise will cross any fixed significance boundary if observed indefinitely. The stopping policy, not the treatment, changes this false-alarm rate.

This A/A simulation, with two identical arms, uses 5,000 repeat experiments, 20 looks, and a fixed seed. Each look adds 100 new incidents per arm by drawing a binomial count, which is the same as 100 independent Bernoulli trials at 40%.

It tests this exact stopping policy, not a universal percentage for every experiment design.

Ordinary fixed-horizon intervals aren't valid for a decision triggered by continuous monitoring. Repeated observation changes the statistical procedure.

Repeated looks aren't the only way to multiply chances. Trying many variants, metrics, or subgroups and reporting whichever interval excludes zero also needs multiplicity control. Predeclare the family of claims and a method such as Bonferroni, or label new subgroup discoveries exploratory and confirm them with fresh data.^{[5]Reference 5NIST/SEMATECH e-Handbook: Bonferroni's Methodhttps://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm}

Why can't a fixed-horizon experiment stop the first time an ordinary 95% interval excludes zero?

Answer

Repeated unplanned looks give random noise more chances to cross the threshold. Use the planned final look or a sequential method designed for repeated monitoring.

Sequential testing and continuous monitoring

Product teams want dashboards that update in real time and allow early stopping when a treatment is winning convincingly or hurting users. Two principled mathematical frameworks make this possible without inflating false alarms:

Alpha spending functions

Group sequential designs allocate the total false-alarm budget $\alpha$ across intermediate looks using an alpha spending function $\alpha(t)$, where $t = n / N_{\text{max}}$ represents the information fraction ($0 \le t \le 1$).

Lan and DeMets introduced flexible spending functions that preserve the overall Type I error regardless of exact look timings. A popular choice is the O'Brien-Fleming spending boundary:

$$\alpha(t) \approx 2 - 2\Phi\left(\frac{z_{\alpha/2}}{\sqrt{t}}\right)$$

Early in the test (small $t$), the boundary requires extreme evidence (for example, $|z| > 4.0$ or $p < 0.0001$) to stop early. As the test approaches the maximum planned sample size ($t \to 1$), the required critical value relaxes toward the standard 1.96 cutoff. This preserves almost the entire nominal power for the final analysis while providing an emergency brake against catastrophic treatment failures.

Mixture sequential probability ratio tests (mSPRT)

When teams demand true continuous monitoring on every event, the mixture Sequential Probability Ratio Test (mSPRT) provides always-valid p-values and confidence sequences.^{[6]Reference 6Peeking at A/B Tests: Why it matters, and what to do about it.https://doi.org/10.1145/3097983.3097992}

Under mSPRT, you construct a test statistic $\Lambda_n$ by integrating the likelihood ratio across a mixing distribution $H(\theta)$ over possible effect sizes:

$$\Lambda_n = \int \prod_{i=1}^n \frac{f_\theta(Y_i)}{f_0(Y_i)} dH(\theta)$$

Under the null hypothesis $H_0$, $\Lambda_n$ is a non-negative martingale with expectation equal to 1 for all $n$. By Ville's inequality:

$$\mathbb{P}\left(\sup_{n \ge 1} \Lambda_n \ge \frac{1}{\alpha}\right) \le \alpha$$

Because Ville's inequality holds uniformly across all sample sizes $n$, the resulting confidence sequences never expire. Product managers can inspect the dashboard at any hour and stop the test whenever the always-valid interval excludes zero, with a mathematical guarantee that false alarms stay capped at $\alpha$.

Reduce noise with pre-experiment data

Power doesn't always require more incidents. CUPED (Controlled-experiment Using Pre-Experiment Data) uses a predictive measurement collected before treatment. For a new incident, that could be a frozen summary of its service's resolution history from before the experiment started. It can't be a prior-period outcome of the same not-yet-created incident.

That signal can remove predictable unit-to-unit variation from the outcome.

Let $Y$ be the experiment outcome and $X$ be a pre-experiment covariate. CUPED forms an adjusted metric:

$$Y_{adjusted} = Y - \theta(X - \bar{X})$$

Because treatment assignment is randomized, the expected value of $X$ is identical in both arms: $\mathbb{E}[X_{treatment} - X_{control}] = 0$. Subtracting $\theta(X - \bar{X})$ leaves the expected treatment difference completely unchanged:

$$\mathbb{E}[Y_{adj, treatment} - Y_{adj, control}] = \mathbb{E}[Y_{treatment} - Y_{control}]$$

Now compute the variance of the adjusted metric:

$$\operatorname{Var}(Y_{adjusted}) = \operatorname{Var}(Y) + \theta^2 \operatorname{Var}(X) - 2\theta \operatorname{Cov}(Y, X)$$

Differentiating with respect to $\theta$ and setting to zero yields the optimal coefficient:

$$\theta^* = \frac{\operatorname{Cov}(Y, X)}{\operatorname{Var}(X)}$$

Plugging $\theta^*$ back into the variance equation produces the variance reduction formula:

$$\operatorname{Var}(Y_{adjusted}) = \operatorname{Var}(Y) \left(1 - \frac{\operatorname{Cov}(Y, X)^2}{\operatorname{Var}(Y)\operatorname{Var}(X)}\right) = \operatorname{Var}(Y)(1 - \rho^2)$$

Here $\rho$ is the Pearson correlation between $X$ and $Y$. Under optimal adjustment, outcome variance is multiplied by $1-\rho^2$.

Deng, Xu, Kohavi, and Walker introduced this approach and reported about 50% variance reduction on Bing experiments, enough to cut runtime or traffic roughly in half. Using the same metric from a one-week pre-period reduced queries-per-user variance by more than 45%.^{[7]Reference 7Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED)https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf}

This simulation gives each independent unit a historical flag available before treatment, then assigns treatment independently. Treat the flag as a simplified precomputed feature, not an earlier response to the experiment. Predict which estimate should have the smaller standard error.

In a finite sample, the raw and adjusted estimates won't be numerically identical. The code estimates one common coefficient from pooled experiment data and uses a large-sample plug-in standard error. Predeclare the adjustment; for richer fitted models, use an appropriate regression-adjusted or cross-fitted estimator and standard errors rather than searching for the adjustment with the biggest lift.

Adjusted outcomes can fall outside zero and one. That's acceptable: they're numerical variance-reduction scores, not new Bernoulli labels. Report the adjusted lift and its uncertainty, and check whether the standard error actually improves.

Never adjust for a quantity treatment could affect, such as latency measured after the rewrite is enabled. Conditioning on that post-treatment value creates collider bias or blocks the causal pathway, wrecking the validity of your test.

Bandits versus fixed A/B tests

When should you run a traditional fixed-horizon A/B test, and when should you deploy a multi-armed bandit algorithm?

This choice centers on the classic exploration versus exploitation tradeoff:

• Fixed A/B testing: Enforces equal 50/50 allocation throughout the entire experiment. It prioritizes pure exploration during the trial to maximize statistical power for causal parameter estimation and guardrail verification. Once the launch gate passes, the winning variant receives 100% of traffic (pure exploitation).
• Multi-armed bandits: Adaptively shift traffic toward whichever variant currently looks best, minimizing cumulative regret $R(T) = \sum_{t=1}^T (\mu^* - \mu_{a_t})$ while the experiment runs.^{[8]Reference 8A Tutorial on Thompson Samplinghttps://arxiv.org/abs/1707.02038}

In Thompson Sampling, each arm maintains a Bayesian posterior over its success probability. For binary resolution, a Beta-Bernoulli conjugate model samples $\theta_k \sim \operatorname{Beta}(\alpha_k, \beta_k)$ on each request and routes to the arm with the highest draw.

The bandit routed 76.8% of incidents to the winning treatment arm, resolving more outages during the test. That efficiency carries an inference penalty:

1. Adaptive sampling bias: Arms that get lucky early receive more traffic, while arms that encounter early bad luck are starved. The sample mean of the winning variant is biased upward.
2. Invalid classical inference: Standard two-sample t-tests and confidence intervals collapse because observations aren't independent and identically distributed.
3. Guardrail blindness: Bandits optimize a single scalar reward. If treatment improves resolution by 4% but quietly triples p95 latency or inflates hallucinations, an unconstrained bandit keeps routing more traffic to the slower, hallucinating model.

Deploy bandits for short-lived opportunities where immediate opportunity cost dominates and long-term scientific learning isn't needed: headline testing, holiday marketing campaigns, or transient routing between external API providers. Use fixed A/B tests for core algorithm changes, model migrations, and architectural updates where guardrails and unconfounded causal conclusions are required.

Interleave pairs to slash prompt noise

In conversational AI and incident management, prompts vary wildly in difficulty. Some incidents describe simple permission lockouts; others describe complex distributed deadlocks. In a standard between-subject A/B test (incident A gets Control, incident B gets Treatment), this prompt difficulty variance adds directly to the standard error of the lift.

Interleaved testing solves this problem by evaluating both models on the exact same input.^{[9]Reference 9Large-scale Validation and Analysis of Interleaved Search Evaluation.https://doi.org/10.1145/2094072.2094078} In search ranking, interleaving blends the candidate lists into a single combined presentation. In LLM evaluation, both models generate candidate answers for the same prompt, and an engineer or automated evaluator judges them head-to-head (as popularized in Chatbot Arena).

By turning an independent two-sample test into a paired comparison, the shared prompt variance cancels out:

$$\operatorname{Var}(Y_T - Y_C) = \operatorname{Var}(Y_T) + \operatorname{Var}(Y_C) - 2\operatorname{Cov}(Y_T, Y_C)$$

When both models evaluate the same prompt difficulty, their scores are strongly positively correlated. The covariance term $2\operatorname{Cov}(Y_T, Y_C)$ subtracts out prompt difficulty noise, drastically shrinking the standard error.

Pairing on the same query slashed outcome variance by 89.5%, shrinking standard error from 0.085 down to 0.027. Chapelle et al. demonstrated that interleaved search evaluation can be 100 to 1,000 times more sensitive than traditional A/B testing.^{[9]Reference 9Large-scale Validation and Analysis of Interleaved Search Evaluation.https://doi.org/10.1145/2094072.2094078}

Treat AI evaluation as a stack

An online experiment answers whether the on-call benefits under live use. It shouldn't be the first time you discover that a treatment emits unsupported rollback claims. Check the treatment in layers, each with a different question:

If you use a large language model as a judge for an offline rubric, treat it as a measured evaluator rather than truth. Pin its model and prompt version, then compare it with human labels on a retained calibration set.

Zheng et al. study LLM judges as approximations to human preference and document position bias, verbosity bias, self-enhancement bias (favoring answers the judge model itself produced), and reasoning limitations.^{[10]Reference 10Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

This lab makes the configuration receipt explicit. A comparison with two different decoder versions is rejected before anybody reads its lift.

Make the launch decision auditable

Now combine the evidence. The brief requires a positive interval, an observed lift of at least +2.0 points, guardrails inside their budgets, and trustworthy traffic.

This last lab prints each condition instead of burying the call in a slide deck. Like the earlier lift tables, it reuses the compact 2,000-incident teaching fixture so you can check every line by hand.

For the locked MDE of +2.0 points, the fuller independent-arm calculation targets about 9,336 incidents per arm before accounting for other design requirements. The smaller fixture below is arithmetic practice, not permission for an unplanned early look. Predict which guardrail could keep a positive lift from shipping.

This gate separates evidence from business value. positive_lift_interval asks whether the interval's lower bound is above zero. observed_lift_threshold asks whether the point estimate clears the predeclared +2.0 point bar. Those checks answer different questions.

It doesn't prove the true lift exceeds +2.0 points: the interval begins at +1.0. A more conservative brief could require the lower bound to clear +2.0 points and would keep this result under investigation.

Guardrails can look safe on a point estimate while their uncertainty still crosses the budget. Our brief already declared the grounding rule: the upper endpoint must stay below +0.5 points. Using the upper end of a two-sided 95% interval corresponds to an approximate one-sided 97.5% bound, not a one-sided 95% bound. Keep that choice fixed rather than switching cutoffs after seeing the result.

For grounding failures as a rate, the unpooled SE is:

$$SE(\widehat{\Delta}_g) = \sqrt{ \frac{\hat{g}_{c}(1 - \hat{g}_{c})}{n_{c}} + \frac{\hat{g}_{t}(1 - \hat{g}_{t})}{n_{t}} }$$

With the teaching fixture, grounding rates are 1.10% for control and 1.15% for treatment, so $\widehat{\Delta}_g = +0.05$ pp. The unpooled SE is about 0.333 pp, which puts the rough 95% interval at about $[-0.60, +0.70]$ pp. Its upper end exceeds the +0.5 pp budget, so grounding_noninferiority fails even though the observed change is small.

For these teaching numbers, the lift and point-estimate checks pass, but the grounding rule keeps the decision at HOLD AND INVESTIGATE. Failing to establish non-inferiority doesn't prove that grounding became worse: the interval includes no change too. We lack enough precision to rule out the unacceptable regression.

The rare-event grounding interval is only a normal approximation with 22 and 23 failures. A real launch should use a preselected method suitable for those counts and audit sampling. Even a passing teaching gate wouldn't resolve the omitted latency and human-page uncertainty rules.

Change treatment latency to 1490 and rerun: resolution still improves, but the speed budget also fails. A treatment doesn't earn launch by winning only its favorite metric.

Practice: break the launch gate

Change one field at a time and predict the gate before rerunning it. These are counterfactual exercises, not permission to edit a live experiment's brief after seeing results:

1. Change treatment latency from 1260 to 1490. Which check fails, and why doesn't the resolution lift override it?
2. Restore latency, then change minimum_ship_lift from 0.02 to 0.05. Which check fails even though the interval still excludes zero?
3. Restore the threshold, then set srm_alert to True. Why should analysis pause before anybody argues about the observed lift?

Expected observations

1. latency_budget fails because treatment is now 310 ms slower than control, beyond the locked 150 ms budget.
2. observed_lift_threshold fails because the observed +4.0 point lift is below the new +5.0 point business threshold. Statistical evidence and launch value answer different questions.
3. The code raises ValueError: Investigate SRM before computing lift. SRM can signal broken assignment, filtering, or logging, so the outcome calculation doesn't run. It isn't merely a red badge added after displaying an untrustworthy lift.

Freeze the experiment receipt

Save the assignment unit and hash rule, metric query and outcome window, planned analysis schedule, sample-ratio-mismatch check, effect estimate with uncertainty, guardrails, and final decision. Those artifacts let a reviewer rerun the gate without reconstructing policy from a dashboard. A trustworthy experiment ends as a reproducible decision record, not a screenshot of a favorable number.